SlideShare a Scribd company logo
1Confidential
Query the App, Not the DB:
“Interactive Queries” in
Apache® Kafka’sTM Streams
API
Matthias J. Sax | Software Engineer
matthias@confluent.io
@MatthiasJSax
2Confidential
Outline
• Kafka Streams in 30 seconds
• What is state and why stateful stream processing?
• Technical Deep Dive on Interactive Queries
• What does IQ provide?
• State handling with Kafka Streams
• How to use IQ?
• Wrap up
3Confidential
Kafka Streams Overview
Simplest way to process data streams with Kafka
• Kafka added Streams API in 0.10.0 release
• Powerful stream processing library, tightly integrated with Kafka
• Easy to use high-level DSL (plus low-level Processor API)
• Rich time semantics
• Event-, ingestion-, processing-time
• Handles out-of-order records
• Powerful: aggregations, joins, windows, tables, stateful processing
• Scalable, elastic, and highly available applications
4Confidential
Kafka Streams Overview
5Confidential
Stateful Stream Processing
6Confidential
Stateful Stream Processing
• What is State?
• Anything your application needs to “remember” beyond the scope of a
single record
• Stateless operators:
• filter, map, flatMap, foreach
• Stateful operator:
• Aggregations (count, min, max, sum, average, …)
• Any window operation
• Joins
• CEP
Almost any non-trivial stream processing application will have state!
7Confidential
Old Fashioned Stateful Stream Processing
• Need to maintain DB
• More moving parts
• More error prone
• No app isolation
• Expensive I/O
8Confidential
Old Fashioned Stateful Stream Processing
Before Kafka Streams L
9Confidential
Kafka Streams: “Cluster to go” – “DB to go”
With Kafka Streams and “Interactive Queries” J
10Confidential
Accessing Application State
• External DB: query database
• Local State: Interactive Queries
• Allows to query the local DB instances
• Kafka Streams application is the database
• Read-Only access (no “external” state updates)
• Works for DSL or Processor API
11Confidential
Accessing Application State – “DB to go”
• Interactive Queries turn Kafka Streams application into
queryable light-weight database
With Kafka Streams and “Interactive Queries” J
12Confidential
Technical Deep Dive
What can Interactive Queries deliver and how to use it?
13Confidential
Remember…
14Confidential
• Light-weight embedded DB
• Local data access
• No RPC, network I/O
Kafka Streams uses Local State
15Confidential
Fault-Tolerant Local State
16Confidential
Scaling Local State
• Topics are scaled via partitions
• Kafka Streams scales via tasks
• 1:1 mapping from input topic partitions to tasks
• State can be sharded based on partitions
• Local DB for each shard
• 1:1 mapping from state shard to task
17Confidential
Scaling Local State
18Confidential
Scaling your Application
19Confidential
Scaling your Stateful Application
20Confidential
key-X 7key-A 5 key-g 4
key-B 7 key-h 2 key-Y 3
How to use Interactive Queries
1. Query local state
2. Discover remote instances
3. Query remote state
21Confidential
Query Local State
key-A 5
key-B 7
get(“key-A”)
22Confidential
Query Local State
Get access to local store via ”store name”
// specify your application logic
KStreamBuilder builder = new KStreamBuilder();
builder.stream(“input-topic”)
.groupByKey()
.count(“counts-store”);
Properties props = new Properties();
// set some configs -- omitted for brevity
// create your application and start it
KafkaStreams streams = new KafkaStreams(builder, props);
streams.start();
// get local store (ie, local shard)
ReadOnlyKeyValueStore<K,V> queryableStore =
streams.store(“counts-store”, QueryableStoreTypes.<K,V>keyValueStore());
23Confidential
• Kafka Streams application instances are started
independently
• What other instances are there?
• Which instances hold which data?
Discover Remote Instances
key-A 5
key-B 7
get(“key-C”)
key-g 4
key-h 2
get(“key-A”)
24Confidential
Discover Remote Instances
• Kafka Streams metadata API to the rescue:
• KafkaStreams#allMetadata()
• KafkaStreams#allMetadataForStore(String storeName)
• KafkaStreams#allMetadataForKey(
String storeName,
K key,
Serializer<K> keySerializer)
• Returns StreamsMetadata object
25Confidential
Discover Remote Instances
• StreamsMetadata object
• Host information (server/port) according to
”application.server” config
• State store names of hosted stores
• TopicPartition mapped to the store (ie, shard)
26Confidential
How to access data of another instance?
key-A 5
key-B 7
get(“key-C”)
key-g 4
key-h 2
get(“key-A”)
???
27Confidential
Query remote state via RPC
• Choose whatever you like:
• Java RMI
• REST
• Apache Thrift
• XML-RPC
• SOAP
• …
Kafka Streams is a library and doesn’t dictate the RPC mechanism.
28Confidential
Query remote state
get(“key-C”)
key-A 5
key-B 7
key-g 4
key-h 2
key-X 7
key-Y 3
1234
1234
1235
get(“key-A”)
<key-A,5>
application.server=host1:1234 application.server=host2:1234
application.server=host2:1235
host1 host2
29Confidential
Before IQ L
With IQ J
Take Home Message
30Confidential
Pros and Cons using Interactive Queries
• Pros:
• There are fewer moving pieces; no external database
• It enables faster and more efficient use of the application state
• It provides better isolation
• It allows for flexibility
• Mix-and-match local an remote DB
• Cons:
• It may involve moving away from a datastore you know and trust
• You might want to scale storage independently of processing
• You might need customized queries, specific to some datastores
31Confidential
Summary
• Interactive Queries is a powerful abstractions that simplifies
stateful stream processing
• There are still cases for which external database/storage
might be a better fit – with IQ you can choose!
• Check out
• Kafka music example: https://github.com/confluentinc/examples/blob/master/kafka-
streams/src/main/java/io/confluent/examples/streams/interactivequeries/kafkamusic/KafkaMusicExa
mple.java
• Blog post: https://www.confluent.io/blog/unifying-stream-processing-and-interactive-queries-in-
apache-kafka/
32Confidential
Thank You
We are hiring!

More Related Content

What's hot (20)

PDF
Data integration with Apache Kafka
confluent
 
PDF
Kafka Summit NYC 2017 Introduction to Kafka Streams with a Real-life Example
confluent
 
PDF
Bravo Six, Going Realtime. Transitioning Activision Data Pipeline to Streamin...
HostedbyConfluent
 
PDF
Analyzing Petabyte Scale Financial Data with Apache Pinot and Apache Kafka | ...
HostedbyConfluent
 
PDF
Intro to AsyncAPI
confluent
 
PDF
Inside Kafka Streams—Monitoring Comcast’s Outside Plant
confluent
 
PDF
Introduction to apache kafka, confluent and why they matter
Paolo Castagna
 
PDF
Deploying Confluent Platform for Production
confluent
 
PDF
Improving Logging Ingestion Quality At Pinterest: Fighting Data Corruption An...
HostedbyConfluent
 
PDF
Capital One Delivers Risk Insights in Real Time with Stream Processing
confluent
 
PPTX
Kafka Summit NYC 2017 - Venice: A Distributed Database on top of Kafka
confluent
 
PPTX
Kafka Summit NYC 2017 - Cloud Native Data Streaming Microservices with Spring...
confluent
 
PDF
ksqlDB: A Stream-Relational Database System
confluent
 
PDF
Stream Processing with Apache Kafka and .NET
confluent
 
PDF
Building Realtim Data Pipelines with Kafka Connect and Spark Streaming
Guozhang Wang
 
PDF
Building Retry Architectures in Kafka with Compacted Topics | Matthew Zhou, V...
HostedbyConfluent
 
PDF
Kafka Summit NYC 2017 - Scalable Real-Time Complex Event Processing @ Uber
confluent
 
PPTX
Kafka Summit NYC 2017 Hanging Out with Your Past Self in VR
confluent
 
PDF
Kafka Summit SF 2017 - Database Streaming at WePay
confluent
 
PDF
Real-world Streaming Architectures
confluent
 
Data integration with Apache Kafka
confluent
 
Kafka Summit NYC 2017 Introduction to Kafka Streams with a Real-life Example
confluent
 
Bravo Six, Going Realtime. Transitioning Activision Data Pipeline to Streamin...
HostedbyConfluent
 
Analyzing Petabyte Scale Financial Data with Apache Pinot and Apache Kafka | ...
HostedbyConfluent
 
Intro to AsyncAPI
confluent
 
Inside Kafka Streams—Monitoring Comcast’s Outside Plant
confluent
 
Introduction to apache kafka, confluent and why they matter
Paolo Castagna
 
Deploying Confluent Platform for Production
confluent
 
Improving Logging Ingestion Quality At Pinterest: Fighting Data Corruption An...
HostedbyConfluent
 
Capital One Delivers Risk Insights in Real Time with Stream Processing
confluent
 
Kafka Summit NYC 2017 - Venice: A Distributed Database on top of Kafka
confluent
 
Kafka Summit NYC 2017 - Cloud Native Data Streaming Microservices with Spring...
confluent
 
ksqlDB: A Stream-Relational Database System
confluent
 
Stream Processing with Apache Kafka and .NET
confluent
 
Building Realtim Data Pipelines with Kafka Connect and Spark Streaming
Guozhang Wang
 
Building Retry Architectures in Kafka with Compacted Topics | Matthew Zhou, V...
HostedbyConfluent
 
Kafka Summit NYC 2017 - Scalable Real-Time Complex Event Processing @ Uber
confluent
 
Kafka Summit NYC 2017 Hanging Out with Your Past Self in VR
confluent
 
Kafka Summit SF 2017 - Database Streaming at WePay
confluent
 
Real-world Streaming Architectures
confluent
 

Similar to Kafka Summit SF 2017 - Query the Application, Not a Database: “Interactive Queries” in Kafka’s Streams API (20)

PPTX
Kick your database_to_the_curb_reston_08_27_19
confluent
 
PPTX
Kick Your Database to the Curb
Bill Bejeck
 
PDF
Introducing Kafka's Streams API
confluent
 
PDF
Interactive Query in Kafka Streams: The Next Generation with Vasiliki Papavas...
HostedbyConfluent
 
PDF
A Tour of Apache Kafka
confluent
 
PDF
Rethinking Stream Processing with Apache Kafka: Applications vs. Clusters, St...
Michael Noll
 
PDF
Apache Kafka as Event-Driven Open Source Streaming Platform (Prague Meetup)
Kai Wähner
 
PPTX
Stream Application Development with Apache Kafka
Matthias J. Sax
 
PPTX
Introducing Apache Kafka's Streams API - Kafka meetup Munich, Jan 25 2017
Michael Noll
 
PPTX
Kafka Streams: The Stream Processing Engine of Apache Kafka
Eno Thereska
 
PDF
Queryable State for Kafka Streamsを使ってみた
Yoshiyasu SAEKI
 
PDF
How to Build Streaming Apps with Confluent II
confluent
 
PPTX
Kafka Streams for Java enthusiasts
Slim Baltagi
 
PDF
Rediscovering the Value of Apache Kafka® in Modern Data Architecture
confluent
 
PDF
Streaming Solutions for Real time problems
Abhishek Gupta
 
PDF
What is Apache Kafka and What is an Event Streaming Platform?
confluent
 
ODP
Stream processing using Kafka
Knoldus Inc.
 
PPTX
Building streaming data applications using Kafka*[Connect + Core + Streams] b...
Data Con LA
 
PDF
Query Your Streaming Data on Kafka using SQL: Why, How, and What
HostedbyConfluent
 
PDF
Etl, esb, mq? no! es Apache Kafka®
confluent
 
Kick your database_to_the_curb_reston_08_27_19
confluent
 
Kick Your Database to the Curb
Bill Bejeck
 
Introducing Kafka's Streams API
confluent
 
Interactive Query in Kafka Streams: The Next Generation with Vasiliki Papavas...
HostedbyConfluent
 
A Tour of Apache Kafka
confluent
 
Rethinking Stream Processing with Apache Kafka: Applications vs. Clusters, St...
Michael Noll
 
Apache Kafka as Event-Driven Open Source Streaming Platform (Prague Meetup)
Kai Wähner
 
Stream Application Development with Apache Kafka
Matthias J. Sax
 
Introducing Apache Kafka's Streams API - Kafka meetup Munich, Jan 25 2017
Michael Noll
 
Kafka Streams: The Stream Processing Engine of Apache Kafka
Eno Thereska
 
Queryable State for Kafka Streamsを使ってみた
Yoshiyasu SAEKI
 
How to Build Streaming Apps with Confluent II
confluent
 
Kafka Streams for Java enthusiasts
Slim Baltagi
 
Rediscovering the Value of Apache Kafka® in Modern Data Architecture
confluent
 
Streaming Solutions for Real time problems
Abhishek Gupta
 
What is Apache Kafka and What is an Event Streaming Platform?
confluent
 
Stream processing using Kafka
Knoldus Inc.
 
Building streaming data applications using Kafka*[Connect + Core + Streams] b...
Data Con LA
 
Query Your Streaming Data on Kafka using SQL: Why, How, and What
HostedbyConfluent
 
Etl, esb, mq? no! es Apache Kafka®
confluent
 
Ad

More from confluent (20)

PDF
Stream Processing Handson Workshop - Flink SQL Hands-on Workshop (Korean)
confluent
 
PPTX
Webinar Think Right - Shift Left - 19-03-2025.pptx
confluent
 
PDF
Migration, backup and restore made easy using Kannika
confluent
 
PDF
Five Things You Need to Know About Data Streaming in 2025
confluent
 
PDF
Data in Motion Tour Seoul 2024 - Keynote
confluent
 
PDF
Data in Motion Tour Seoul 2024 - Roadmap Demo
confluent
 
PDF
From Stream to Screen: Real-Time Data Streaming to Web Frontends with Conflue...
confluent
 
PDF
Confluent per il settore FSI: Accelerare l'Innovazione con il Data Streaming...
confluent
 
PDF
Data in Motion Tour 2024 Riyadh, Saudi Arabia
confluent
 
PDF
Build a Real-Time Decision Support Application for Financial Market Traders w...
confluent
 
PDF
Strumenti e Strategie di Stream Governance con Confluent Platform
confluent
 
PDF
Compose Gen-AI Apps With Real-Time Data - In Minutes, Not Weeks
confluent
 
PDF
Building Real-Time Gen AI Applications with SingleStore and Confluent
confluent
 
PDF
Unlocking value with event-driven architecture by Confluent
confluent
 
PDF
Il Data Streaming per un’AI real-time di nuova generazione
confluent
 
PDF
Unleashing the Future: Building a Scalable and Up-to-Date GenAI Chatbot with ...
confluent
 
PDF
Break data silos with real-time connectivity using Confluent Cloud Connectors
confluent
 
PDF
Building API data products on top of your real-time data infrastructure
confluent
 
PDF
Speed Wins: From Kafka to APIs in Minutes
confluent
 
PDF
Evolving Data Governance for the Real-time Streaming and AI Era
confluent
 
Stream Processing Handson Workshop - Flink SQL Hands-on Workshop (Korean)
confluent
 
Webinar Think Right - Shift Left - 19-03-2025.pptx
confluent
 
Migration, backup and restore made easy using Kannika
confluent
 
Five Things You Need to Know About Data Streaming in 2025
confluent
 
Data in Motion Tour Seoul 2024 - Keynote
confluent
 
Data in Motion Tour Seoul 2024 - Roadmap Demo
confluent
 
From Stream to Screen: Real-Time Data Streaming to Web Frontends with Conflue...
confluent
 
Confluent per il settore FSI: Accelerare l'Innovazione con il Data Streaming...
confluent
 
Data in Motion Tour 2024 Riyadh, Saudi Arabia
confluent
 
Build a Real-Time Decision Support Application for Financial Market Traders w...
confluent
 
Strumenti e Strategie di Stream Governance con Confluent Platform
confluent
 
Compose Gen-AI Apps With Real-Time Data - In Minutes, Not Weeks
confluent
 
Building Real-Time Gen AI Applications with SingleStore and Confluent
confluent
 
Unlocking value with event-driven architecture by Confluent
confluent
 
Il Data Streaming per un’AI real-time di nuova generazione
confluent
 
Unleashing the Future: Building a Scalable and Up-to-Date GenAI Chatbot with ...
confluent
 
Break data silos with real-time connectivity using Confluent Cloud Connectors
confluent
 
Building API data products on top of your real-time data infrastructure
confluent
 
Speed Wins: From Kafka to APIs in Minutes
confluent
 
Evolving Data Governance for the Real-time Streaming and AI Era
confluent
 
Ad

Recently uploaded (20)

PPTX
Alan Turing - life and importance for all of us now
Pedro Concejero
 
PPTX
OCS353 DATA SCIENCE FUNDAMENTALS- Unit 1 Introduction to Data Science
A R SIVANESH M.E., (Ph.D)
 
PDF
WD2(I)-RFQ-GW-1415_ Shifting and Filling of Sand in the Pond at the WD5 Area_...
ShahadathHossain23
 
PDF
Bachelor of information technology syll
SudarsanAssistantPro
 
PPTX
原版一样(EC Lille毕业证书)法国里尔中央理工学院毕业证补办
Taqyea
 
PDF
mbse_An_Introduction_to_Arcadia_20150115.pdf
henriqueltorres1
 
PPT
New_school_Engineering_presentation_011707.ppt
VinayKumar304579
 
PDF
SERVERLESS PERSONAL TO-DO LIST APPLICATION
anushaashraf20
 
PPTX
darshai cross section and river section analysis
muk7971
 
PPTX
Knowledge Representation : Semantic Networks
Amity University, Patna
 
PPTX
DATA BASE MANAGEMENT AND RELATIONAL DATA
gomathisankariv2
 
PPTX
Final Major project a b c d e f g h i j k l m
bharathpsnab
 
PPTX
Mechanical Design of shell and tube heat exchangers as per ASME Sec VIII Divi...
shahveer210504
 
PDF
Digital water marking system project report
Kamal Acharya
 
PPTX
2025 CGI Congres - Surviving agile v05.pptx
Derk-Jan de Grood
 
PDF
NTPC PATRATU Summer internship report.pdf
hemant03701
 
PDF
3rd International Conference on Machine Learning and IoT (MLIoT 2025)
ClaraZara1
 
PDF
Halide Perovskites’ Multifunctional Properties: Coordination Engineering, Coo...
TaameBerhe2
 
PPTX
fatigue in aircraft structures-221113192308-0ad6dc8c.pptx
aviatecofficial
 
PPTX
Numerical-Solutions-of-Ordinary-Differential-Equations.pptx
SAMUKTHAARM
 
Alan Turing - life and importance for all of us now
Pedro Concejero
 
OCS353 DATA SCIENCE FUNDAMENTALS- Unit 1 Introduction to Data Science
A R SIVANESH M.E., (Ph.D)
 
WD2(I)-RFQ-GW-1415_ Shifting and Filling of Sand in the Pond at the WD5 Area_...
ShahadathHossain23
 
Bachelor of information technology syll
SudarsanAssistantPro
 
原版一样(EC Lille毕业证书)法国里尔中央理工学院毕业证补办
Taqyea
 
mbse_An_Introduction_to_Arcadia_20150115.pdf
henriqueltorres1
 
New_school_Engineering_presentation_011707.ppt
VinayKumar304579
 
SERVERLESS PERSONAL TO-DO LIST APPLICATION
anushaashraf20
 
darshai cross section and river section analysis
muk7971
 
Knowledge Representation : Semantic Networks
Amity University, Patna
 
DATA BASE MANAGEMENT AND RELATIONAL DATA
gomathisankariv2
 
Final Major project a b c d e f g h i j k l m
bharathpsnab
 
Mechanical Design of shell and tube heat exchangers as per ASME Sec VIII Divi...
shahveer210504
 
Digital water marking system project report
Kamal Acharya
 
2025 CGI Congres - Surviving agile v05.pptx
Derk-Jan de Grood
 
NTPC PATRATU Summer internship report.pdf
hemant03701
 
3rd International Conference on Machine Learning and IoT (MLIoT 2025)
ClaraZara1
 
Halide Perovskites’ Multifunctional Properties: Coordination Engineering, Coo...
TaameBerhe2
 
fatigue in aircraft structures-221113192308-0ad6dc8c.pptx
aviatecofficial
 
Numerical-Solutions-of-Ordinary-Differential-Equations.pptx
SAMUKTHAARM
 

Kafka Summit SF 2017 - Query the Application, Not a Database: “Interactive Queries” in Kafka’s Streams API

  • 1. 1Confidential Query the App, Not the DB: “Interactive Queries” in Apache® Kafka’sTM Streams API Matthias J. Sax | Software Engineer [email protected] @MatthiasJSax
  • 2. 2Confidential Outline • Kafka Streams in 30 seconds • What is state and why stateful stream processing? • Technical Deep Dive on Interactive Queries • What does IQ provide? • State handling with Kafka Streams • How to use IQ? • Wrap up
  • 3. 3Confidential Kafka Streams Overview Simplest way to process data streams with Kafka • Kafka added Streams API in 0.10.0 release • Powerful stream processing library, tightly integrated with Kafka • Easy to use high-level DSL (plus low-level Processor API) • Rich time semantics • Event-, ingestion-, processing-time • Handles out-of-order records • Powerful: aggregations, joins, windows, tables, stateful processing • Scalable, elastic, and highly available applications
  • 6. 6Confidential Stateful Stream Processing • What is State? • Anything your application needs to “remember” beyond the scope of a single record • Stateless operators: • filter, map, flatMap, foreach • Stateful operator: • Aggregations (count, min, max, sum, average, …) • Any window operation • Joins • CEP Almost any non-trivial stream processing application will have state!
  • 7. 7Confidential Old Fashioned Stateful Stream Processing • Need to maintain DB • More moving parts • More error prone • No app isolation • Expensive I/O
  • 8. 8Confidential Old Fashioned Stateful Stream Processing Before Kafka Streams L
  • 9. 9Confidential Kafka Streams: “Cluster to go” – “DB to go” With Kafka Streams and “Interactive Queries” J
  • 10. 10Confidential Accessing Application State • External DB: query database • Local State: Interactive Queries • Allows to query the local DB instances • Kafka Streams application is the database • Read-Only access (no “external” state updates) • Works for DSL or Processor API
  • 11. 11Confidential Accessing Application State – “DB to go” • Interactive Queries turn Kafka Streams application into queryable light-weight database With Kafka Streams and “Interactive Queries” J
  • 12. 12Confidential Technical Deep Dive What can Interactive Queries deliver and how to use it?
  • 14. 14Confidential • Light-weight embedded DB • Local data access • No RPC, network I/O Kafka Streams uses Local State
  • 16. 16Confidential Scaling Local State • Topics are scaled via partitions • Kafka Streams scales via tasks • 1:1 mapping from input topic partitions to tasks • State can be sharded based on partitions • Local DB for each shard • 1:1 mapping from state shard to task
  • 20. 20Confidential key-X 7key-A 5 key-g 4 key-B 7 key-h 2 key-Y 3 How to use Interactive Queries 1. Query local state 2. Discover remote instances 3. Query remote state
  • 21. 21Confidential Query Local State key-A 5 key-B 7 get(“key-A”)
  • 22. 22Confidential Query Local State Get access to local store via ”store name” // specify your application logic KStreamBuilder builder = new KStreamBuilder(); builder.stream(“input-topic”) .groupByKey() .count(“counts-store”); Properties props = new Properties(); // set some configs -- omitted for brevity // create your application and start it KafkaStreams streams = new KafkaStreams(builder, props); streams.start(); // get local store (ie, local shard) ReadOnlyKeyValueStore<K,V> queryableStore = streams.store(“counts-store”, QueryableStoreTypes.<K,V>keyValueStore());
  • 23. 23Confidential • Kafka Streams application instances are started independently • What other instances are there? • Which instances hold which data? Discover Remote Instances key-A 5 key-B 7 get(“key-C”) key-g 4 key-h 2 get(“key-A”)
  • 24. 24Confidential Discover Remote Instances • Kafka Streams metadata API to the rescue: • KafkaStreams#allMetadata() • KafkaStreams#allMetadataForStore(String storeName) • KafkaStreams#allMetadataForKey( String storeName, K key, Serializer<K> keySerializer) • Returns StreamsMetadata object
  • 25. 25Confidential Discover Remote Instances • StreamsMetadata object • Host information (server/port) according to ”application.server” config • State store names of hosted stores • TopicPartition mapped to the store (ie, shard)
  • 26. 26Confidential How to access data of another instance? key-A 5 key-B 7 get(“key-C”) key-g 4 key-h 2 get(“key-A”) ???
  • 27. 27Confidential Query remote state via RPC • Choose whatever you like: • Java RMI • REST • Apache Thrift • XML-RPC • SOAP • … Kafka Streams is a library and doesn’t dictate the RPC mechanism.
  • 28. 28Confidential Query remote state get(“key-C”) key-A 5 key-B 7 key-g 4 key-h 2 key-X 7 key-Y 3 1234 1234 1235 get(“key-A”) <key-A,5> application.server=host1:1234 application.server=host2:1234 application.server=host2:1235 host1 host2
  • 29. 29Confidential Before IQ L With IQ J Take Home Message
  • 30. 30Confidential Pros and Cons using Interactive Queries • Pros: • There are fewer moving pieces; no external database • It enables faster and more efficient use of the application state • It provides better isolation • It allows for flexibility • Mix-and-match local an remote DB • Cons: • It may involve moving away from a datastore you know and trust • You might want to scale storage independently of processing • You might need customized queries, specific to some datastores
  • 31. 31Confidential Summary • Interactive Queries is a powerful abstractions that simplifies stateful stream processing • There are still cases for which external database/storage might be a better fit – with IQ you can choose! • Check out • Kafka music example: https://github.com/confluentinc/examples/blob/master/kafka- streams/src/main/java/io/confluent/examples/streams/interactivequeries/kafkamusic/KafkaMusicExa mple.java • Blog post: https://www.confluent.io/blog/unifying-stream-processing-and-interactive-queries-in- apache-kafka/