SlideShare a Scribd company logo

Introducing Kafka's Streams API

confluent
confluent

The document introduces Apache Kafka's Streams API for stream processing. Some key points covered include: - The Streams API allows building stream processing applications without needing a separate cluster, providing an elastic, scalable, and fault-tolerant processing engine. - It integrates with existing Kafka deployments and supports both stateful and stateless computations on data in Kafka topics. - Applications built with the Streams API are standard Java applications that run on client machines and leverage Kafka for distributed, parallel processing and fault tolerance via state stores in Kafka.

Software
1 of 118
Downloaded 191 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
1Confidential Introducing Kafka’s Streams API Stream processing made simple Target audience: technical staff, developers, architects Expected duration for full deck: 45 minutes
2Confidential 0.10 Data processing (Streams API) 0.9 Data integration (Connect API) Intra-cluster replication 0.8 Apache Kafka: birthed as a messaging system, now a streaming platform 2012 2014 2015 2016 2017 Cluster mirroring, data compression 0.7 2013
3Confidential Kafka’s Streams API: the easiest way to process data in Apache Kafka Key Benefits of Apache Kafka’s Streams API • Build Apps, Not Clusters: no additional cluster required • Cluster to go: elastic, scalable, distributed, fault-tolerant, secure • Database to go: tables, local state, interactive queries • Equally viable for S / M / L / XL / XXL use cases • “Runs Everywhere”: integrates with your existing deployment strategies such as containers, automation, cloud Part of open source Apache Kafka, introduced in 0.10+ • Powerful client library to build stream processing apps • Apps are standard Java applications that run on client machines • https://github.com/apache/kafka/tree/trunk/streams Streams API Your App Kafka Cluster
4Confidential Kafka’s Streams API: Unix analogy $  cat  <  in.txt | grep “apache”  | tr a-­‐z  A-­‐Z  > out.txt Kafka  Cluster Connect  API Streams  API
5Confidential Streams API in the context of Kafka Streams API Your App Kafka Cluster ConnectAPI ConnectAPI OtherSystems OtherSystems
6Confidential When to use Kafka’s Streams API • Mainstream Application Development • To build core business applications • Microservices • Fast Data apps for small and big data • Reactive applications • Continuous queries and transformations • Event-triggered processes • The “T” in ETL • <and more> Use case examples • Real-time monitoring and intelligence • Customer 360-degree view • Fraud detection • Location-based marketing • Fleet management • <and more>
7Confidential Some public use cases in the wild & external articles • Applying Kafka’s Streams API for internal message delivery pipeline at LINE Corp. • http://developers.linecorp.com/blog/?p=3960 • Kafka Streams in production at LINE, a social platform based in Japan with 220+ million users • Microservices and reactive applications at Capital One • https://speakerdeck.com/bobbycalderwood/commander-decoupled-immutable-rest-apis-with-kafka-streams • User behavior analysis • https://timothyrenner.github.io/engineering/2016/08/11/kafka-streams-not-looking-at-facebook.html • Containerized Kafka Streams applications in Scala • https://www.madewithtea.com/processing-tweets-with-kafka-streams.html • Geo-spatial data analysis • http://www.infolace.com/blog/2016/07/14/simple-spatial-windowing-with-kafka-streams/ • Language classification with machine learning • https://dzone.com/articles/machine-learning-with-kafka-streams
8Confidential Do more with less
9Confidential Architecture comparison: use case example Real-time dashboard for security monitoring “Which of my data centers are under attack?”
10Confidential Architecture comparison: use case example Other App Dashboard Frontend App Other App 1 Capture business events in Kafka 2 Must process events with separate cluster (e.g. Spark) 4 Other apps access latest results by querying these DBs 3 Must share latest results through separate systems (e.g. MySQL) Before: Undue complexity, heavy footprint, many technologies, split ownership with conflicting priorities Your “Job” Other App Dashboard Frontend App Other App 1 Capture business events in Kafka 2 Process events with standard Java apps that use Kafka Streams 3 Now other apps can directly query the latest results With Kafka Streams: simplified, app-centric architecture, puts app owners in control Kafka Streams Your App
11Confidential
12Confidential
13Confidential How do I install the Streams API? • There is and there should be no “installation” – Build Apps, Not Clusters! • It’s a library. Add it to your app like any other library. <dependency> <groupId>org.apache.kafka</groupId> <artifactId>kafka-­‐streams</artifactId> <version>0.10.1.1</version> </dependency>
14Confidential “But wait a minute – where’s THE CLUSTER to process the data?” • No cluster needed – Build Apps, Not Clusters! • Unlearn bad habits: “do cool stuff with data ≠ must have cluster” Ok. Ok. Ok.
15Confidential Organizational benefits: decouple teams and roadmaps, scale people
16Confidential Organizational benefits: decouple teams and roadmaps, scale people Infrastructure Team (Kafka as a shared, multi-tenant service) Fraud detection app Payments team Recommenda tions app Mobile team Security alerts app Operations team ...more apps... ...
17Confidential How do I package, deploy, monitor my apps? How do I …? • Whatever works for you. Stick to what you/your company think is the best way. • No magic needed. • Why? Because an app that uses the Streams API is…a normal Java app.
18Confidential Available APIs
19Confidential The API is but the tip of the iceberg API,  coding Org.  processes Reality™ Deployment Operations Security … Architecture Debugging
20Confidential • API option 1: DSL (declarative) KStream<Integer,  Integer>  input  = builder.stream("numbers-­‐topic"); //  Stateless  computation KStream<Integer,  Integer>  doubled  = input.mapValues(v  -­‐>  v  *  2); //  Stateful  computation KTable<Integer,  Integer>  sumOfOdds =  input .filter((k,v)  -­‐>  v  %  2  !=  0) .selectKey((k,  v)  -­‐>  1) .groupByKey() .reduce((v1,  v2)  -­‐>  v1  +  v2,  "sum-­‐of-­‐odds"); The preferred API for most use cases. Particularly appeals to: • Fans of Scala, functional programming • Users familiar with e.g. Spark
21Confidential • API option 2: Processor API (imperative) class  PrintToConsoleProcessor implements  Processor<K,  V>  { @Override public  void  init(ProcessorContext context)  {} @Override void  process(K  key,  V  value)  {   System.out.println("Got  value  "  +  value);   } @Override void  punctuate(long  timestamp)  {} @Override void  close()  {} } Full flexibility but more manual work Appeals to: • Users who require functionality that is not yet available in the DSL • Users familiar with e.g. Storm, Samza • Still, check out the DSL!
22Confidential When to use Kafka Streams vs. Kafka’s “normal” consumer clients Kafka Streams • Basically all the time • Basically all the time • Basically all the time • Basically all the time • Basically all the time • Basically all the time • Basically all the time • Basically all the time • Basically all the time • Basically all the time • Basically all the time Kafka consumer clients (Java, C/C++, Python, Go, …) • When you must interact with Kafka at a very low level and/or in a very special way • Example: When integrating your own stream processing tool (Spark, Storm) with Kafka.
23Confidential Code comparison Featuring Kafka with Streams API <-> Spark Streaming
24Confidential ”My WordCount is better than your WordCount” (?) Kafka Spark These isolated code snippets are nice (and actually quite similar) but they are not very meaningful. In practice, we also need to read data from somewhere, write data back to somewhere, etc.– but we can see none of this here.
25Confidential WordCount in Kafka Word Count
26Confidential Compared to: WordCount in Spark 2.0 1 2 3 Runtime model leaks into processing logic (here: interfacing from Spark with Kafka)
27Confidential Compared to: WordCount in Spark 2.0 4 5 Runtime model leaks into processing logic (driver vs. executors)
28Confidential Key concepts
29Confidential Key concepts
30Confidential Key concepts
31Confidential Key concepts Kafka Core Kafka Streams
32Confidential Streams and Tables Stream Processing meets Databases
33Confidential
34Confidential
35Confidential Key observation: close relationship between Streams and Tables http://www.confluent.io/blog/introducing-kafka-streams-stream-processing-made-simple http://docs.confluent.io/current/streams/concepts.html#duality-of-streams-and-tables
36Confidential
37Confidential Example: Streams and Tables in Kafka Word Count hello 2 kafka 1 world 1 … …
38Confidential
39Confidential
40Confidential
41Confidential
42Confidential Example: continuously compute current users per geo-region 4 7 5 3 2 8 4 7 6 3 2 7 Alice Real-time dashboard “How many users younger than 30y, per region?” alice Europe user-locations alice Asia, 25y, … bob Europe, 46y, … … … alice Europe, 25y, … bob Europe, 46y, … … … -1 +1 user-locations (mobile team) user-prefs (web team)
43Confidential Example: continuously compute current users per geo-region KTable<UserId,  Location>  userLocations =  builder.table(“user-­‐locations-­‐topic”); KTable<UserId,  Prefs>        userPrefs =  builder.table(“user-­‐preferences-­‐topic”);
44Confidential Example: continuously compute current users per geo-region alice Europe user-locations alice Asia, 25y, … bob Europe, 46y, … … … alice Europe, 25y, … bob Europe, 46y, … … … KTable<UserId,  Location>  userLocations =  builder.table(“user-­‐locations-­‐topic”); KTable<UserId,  Prefs>        userPrefs =  builder.table(“user-­‐preferences-­‐topic”); //  Merge  into  detailed  user  profiles  (continuously  updated) KTable<UserId,  UserProfile>  userProfiles = userLocations.join(userPrefs,  (loc,  prefs)  -­‐>  new  UserProfile(loc,  prefs)); KTable userProfilesKTable userProfiles
45Confidential Example: continuously compute current users per geo-region KTable<UserId,  Location>  userLocations =  builder.table(“user-­‐locations-­‐topic”); KTable<UserId,  Prefs>        userPrefs =  builder.table(“user-­‐preferences-­‐topic”); //  Merge  into  detailed  user  profiles  (continuously  updated) KTable<UserId,  UserProfile>  userProfiles = userLocations.join(userPrefs,  (loc,  prefs)  -­‐>  new  UserProfile(loc,  prefs)); //  Compute  per-­‐region  statistics  (continuously  updated) KTable<UserId,  Long>  usersPerRegion =  userProfiles .filter((userId,  profile)    -­‐>  profile.age <  30) .groupBy((userId,  profile)  -­‐>  profile.location) .count(); alice Europe user-locations Africa 3 … … Asia 8 Europe 5 Africa 3 … … Asia 7 Europe 6 KTable usersPerRegion KTable usersPerRegion
46Confidential Example: continuously compute current users per geo-region 4 7 5 3 2 8 4 7 6 3 2 7 Alice Real-time dashboard “How many users younger than 30y, per region?” alice Europe user-locations alice Asia, 25y, … bob Europe, 46y, … … … alice Europe, 25y, … bob Europe, 46y, … … … -1 +1 user-locations (mobile team) user-prefs (web team)
47Confidential Streams meet Tables – in the DSL
48Confidential Streams meet Tables • Most use cases for stream processing require both Streams and Tables • Essential for any stateful computations • Kafka ships with first-class support for Streams and Tables • Scalability, fault tolerance, efficient joins and aggregations, … • Benefits include: simplified architectures, less moving pieces, less Do-It-Yourself work
49Confidential Key features
50Confidential Key features in 0.10 • Native, 100%-compatible Kafka integration
51Confidential Native, 100% compatible Kafka integration Read  from  Kafka Write  to  Kafka
52Confidential Key features in 0.10 • Native, 100%-compatible Kafka integration • Secure stream processing using Kafka’s security features
53Confidential Secure stream processing with the Streams API • Your applications can leverage all client-side security features in Apache Kafka • Security features include: • Encrypting data-in-transit between applications and Kafka clusters • Authenticating applications against Kafka clusters (“only some apps may talk to the production cluster”) • Authorizing application against Kafka clusters (“only some apps may read data from sensitive topics”)
54Confidential Configuring security settings • In general, you can configure both Kafka Streams plus the underlying Kafka clients in your apps
55Confidential Configuring security settings • Example: encrypting data-in-transit + client authentication to Kafka cluster Full demo application at https://github.com/confluentinc/examples
56Confidential Key features in 0.10 • Native, 100%-compatible Kafka integration • Secure stream processing using Kafka’s security features • Elastic and highly scalable • Fault-tolerant
57Confidential
58Confidential
59Confidential
60Confidential Key features in 0.10 • Native, 100%-compatible Kafka integration • Secure stream processing using Kafka’s security features • Elastic and highly scalable • Fault-tolerant • Stateful and stateless computations
61Confidential Stateful computations • Stateful computations like aggregations (e.g. counting), joins, or windowing require state • State stores are the backbone of state management • … are local for best performance • … are backed up to Kafka for elasticity and for fault-tolerance • ... are per stream task for isolation – think: share-nothing • Pluggable storage engines • Default: RocksDB (a key-value store) to allow for local state that is larger than available RAM • You can also use your own, custom storage engine • From the user perspective: • DSL: no need to worry about anything, state management is automatically being done for you • Processor API: direct access to state stores – very flexible but more manual work
62Confidential
63Confidential
64Confidential
65Confidential
66Confidential Use case: real-time, distributed joins at large scale
67Confidential Use case: real-time, distributed joins at large scale
68Confidential Use case: real-time, distributed joins at large scale
69Confidential Stateful computations • Use the Processor API to interact directly with state stores Get  the  store Use  the  store
70Confidential Key features in 0.10 • Native, 100%-compatible Kafka integration • Secure stream processing using Kafka’s security features • Elastic and highly scalable • Fault-tolerant • Stateful and stateless computations • Interactive queries
71Confidential
72Confidential Interactive Queries: architecture comparison Kafka Streams App App App App 1 Capture business events in Kafka 2 Process the events with Kafka Streams 4 Other apps query external systems for latest results ! Must use external systems to share latest results App App App 1 Capture business events in Kafka 2 Process the events with Kafka Streams 3 Now other apps can directly query the latest results Before (0.10.0) After (0.10.1): simplified, more app-centric architecture Kafka Streams App
73Confidential Key features in 0.10 • Native, 100%-compatible Kafka integration • Secure stream processing using Kafka’s security features • Elastic and highly scalable • Fault-tolerant • Stateful and stateless computations • Interactive queries • Time model
74Confidential Time
75Confidential Time A C B
76Confidential Time • You configure the desired time semantics through timestamp extractors • Default extractor yields event-time semantics • Extracts embedded timestamps of Kafka messages (introduced in v0.10)
77Confidential Key features in 0.10 • Native, 100%-compatible Kafka integration • Secure stream processing using Kafka’s security features • Elastic and highly scalable • Fault-tolerant • Stateful and stateless computations • Interactive queries • Time model • Windowing
78Confidential Windowing • Group events in a stream using time-based windows • Use case examples: • Time-based analysis of ad impressions (”number of ads clicked in the past hour”) • Monitoring statistics of telemetry data (“1min/5min/15min averages”) Input data, where colors represent different users events Rectangles denote different event-time windows processing-time event-time windowing alice bob dave
79Confidential Windowing in the DSL TimeWindows.of(3000) TimeWindows.of(3000).advanceBy(1000)
80Confidential Key features in 0.10 • Native, 100%-compatible Kafka integration • Secure stream processing using Kafka’s security features • Elastic and highly scalable • Fault-tolerant • Stateful and stateless computations • Interactive queries • Time model • Windowing • Supports late-arriving and out-of-order data
81Confidential Out-of-order and late-arriving data • Is very common in practice, not a rare corner case • Related to time model discussion
82Confidential Out-of-order and late-arriving data: example when this will happen Users with mobile phones enter airplane, lose Internet connectivity Emails are being written during the 10h flight Internet connectivity is restored, phones will send queued emails now
83Confidential Out-of-order and late-arriving data • Is very common in practice, not a rare corner case • Related to time model discussion • We want control over how out-of-order data is handled, and handling must be efficient • Example: We process data in 5-minute windows, e.g. compute statistics • Option A: When event arrives 1 minute late: update the original result! • Option B: When event arrives 2 hours late: discard it!
84Confidential Key features in 0.10 • Native, 100%-compatible Kafka integration • Secure stream processing using Kafka’s security features • Elastic and highly scalable • Fault-tolerant • Stateful and stateless computations • Interactive queries • Time model • Windowing • Supports late-arriving and out-of-order data • Millisecond processing latency, no micro-batching • At-least-once processing guarantees (exactly-once is in the works as we speak)
85Confidential Roadmap Outlook
86Confidential Roadmap outlook for Kafka Streams • Exactly-Once processing semantics • Unified API for real-time processing and “batch” processing • Global KTables • Session windows • … and more …
87Confidential Wrapping Up
88Confidential Where to go from here • Kafka Streams is available in Confluent Platform 3.1 and in Apache Kafka 0.10.1 • http://www.confluent.io/download • Kafka Streams demos: https://github.com/confluentinc/examples • Java 7, Java 8+ with lambdas, and Scala • WordCount, Interactive Queries, Joins, Security, Windowing, Avro integration, … • Confluent documentation: http://docs.confluent.io/current/streams/ • Quickstart, Concepts, Architecture, Developer Guide, FAQ • Recorded talks • Introduction to Kafka Streams: http://www.youtube.com/watch?v=o7zSLNiTZbA • Application Development and Data in the Emerging World of Stream Processing (higher level talk): https://www.youtube.com/watch?v=JQnNHO5506w
89Confidential Thank You
90Confidential Appendix: Streams and Tables A closer look
91Confidential Motivating example: continuously compute current users per geo-region 4 7 5 3 2 8 Real-time dashboard “How many users younger than 30y, per region?” alice Asia, 25y, … bob Europe, 46y, … … … user-locations (mobile team) user-prefs (web team)
92Confidential Motivating example: continuously compute current users per geo-region 4 7 5 3 2 8 Real-time dashboard “How many users younger than 30y, per region?” alice Europe user-locations alice Asia, 25y, … bob Europe, 46y, … … … user-locations (mobile team) user-prefs (web team)
93Confidential Motivating example: continuously compute current users per geo-region 4 7 5 3 2 8 Real-time dashboard “How many users younger than 30y, per region?” alice Europe user-locations user-locations (mobile team) user-prefs (web team) alice Asia, 25y, … bob Europe, 46y, … … … alice Europe, 25y, … bob Europe, 46y, … … …
94Confidential Motivating example: continuously compute current users per geo-region 4 7 5 3 2 8 4 7 6 3 2 7 Alice Real-time dashboard “How many users younger than 30y, per region?” alice Europe user-locations alice Asia, 25y, … bob Europe, 46y, … … … alice Europe, 25y, … bob Europe, 46y, … … … -1 +1 user-locations (mobile team) user-prefs (web team)
95Confidential Same data, but different use cases require different interpretations alice San Francisco alice New York City alice Rio de Janeiro alice Sydney alice Beijing alice Paris alice Berlin
96Confidential Same data, but different use cases require different interpretations alice San Francisco alice New York City alice Rio de Janeiro alice Sydney alice Beijing alice Paris alice Berlin Use  case  1:  Frequent  traveler  status? Use  case  2:  Current  location?
97Confidential Same data, but different use cases require different interpretations “Alice has been to SFO, NYC, Rio, Sydney, Beijing, Paris, and finally Berlin.” “Alice is in SFO, NYC, Rio, Sydney, Beijing, Paris, Berlin right now.” ⚑ ⚑ ⚑⚑ ⚑ ⚑ ⚑ ⚑ ⚑ ⚑⚑ ⚑ ⚑ ⚑ Use  case  1:  Frequent  traveler  status? Use  case  2:  Current  location?
98Confidential Same data, but different use cases require different interpretations alice San Francisco alice New York City alice Rio de Janeiro alice Sydney alice Beijing alice Paris alice Berlin Use  case  1:  Frequent  traveler  status? Use  case  2:  Current  location? ⚑ ⚑ ⚑⚑ ⚑ ⚑⚑ ⚑
99Confidential Same data, but different use cases require different interpretations alice San Francisco alice New York City alice Rio de Janeiro alice Sydney alice Beijing alice Paris alice Berlin Use  case  1:  Frequent  traveler  status? Use  case  2:  Current  location? ⚑ ⚑ ⚑⚑ ⚑ ⚑⚑ ⚑
100Confidential Same data, but different use cases require different interpretations alice San Francisco alice New York City alice Rio de Janeiro alice Sydney alice Beijing alice Paris alice Berlin Use  case  1:  Frequent  traveler  status? Use  case  2:  Current  location? ⚑ ⚑ ⚑⚑ ⚑ ⚑⚑ ⚑
101Confidential Streams meet Tables record stream When you need… so that the topic is interpreted as a All the values of a key KStream then you’d read the Kafka topic into a Example All the places Alice has ever been to with messages interpreted as INSERT (append)
102Confidential Streams meet Tables record stream changelog stream When you need… so that the topic is interpreted as a All the values of a key Latest value of a key KStream KTable then you’d read the Kafka topic into a Example All the places Alice has ever been to Where Alice is right now with messages interpreted as INSERT (append) UPSERT (overwrite existing)
103Confidential Same data, but different use cases require different interpretations “Alice has been to SFO, NYC, Rio, Sydney, Beijing, Paris, and finally Berlin.” “Alice is in SFO, NYC, Rio, Sydney, Beijing, Paris, Berlin right now.” ⚑ ⚑ ⚑⚑ ⚑ ⚑ ⚑ ⚑ ⚑ ⚑⚑ ⚑ ⚑ ⚑ Use  case  1:  Frequent  traveler  status? Use  case  2:  Current  location? KStream KTable
104Confidential Motivating example: continuously compute current users per geo-region 4 7 5 3 2 8 4 7 6 3 2 7 Alice Real-time dashboard “How many users younger than 30y, per region?” alice Europe user-locations alice Asia, 25y, … bob Europe, 46y, … … … alice Europe, 25y, … bob Europe, 46y, … … … -1 +1 user-locations (mobile team) user-prefs (web team)
105Confidential Motivating example: continuously compute current users per geo-region KTable<UserId,  Location>  userLocations =  builder.table(“user-­‐locations-­‐topic”); KTable<UserId,  Prefs>        userPrefs =  builder.table(“user-­‐preferences-­‐topic”);
106Confidential Motivating example: continuously compute current users per geo-region alice Europe user-locations alice Asia, 25y, … bob Europe, 46y, … … … alice Europe, 25y, … bob Europe, 46y, … … … KTable<UserId,  Location>  userLocations =  builder.table(“user-­‐locations-­‐topic”); KTable<UserId,  Prefs>        userPrefs =  builder.table(“user-­‐preferences-­‐topic”); //  Merge  into  detailed  user  profiles  (continuously  updated) KTable<UserId,  UserProfile>  userProfiles = userLocations.join(userPrefs,  (loc,  prefs)  -­‐>  new  UserProfile(loc,  prefs)); KTable userProfilesKTable userProfiles
107Confidential Motivating example: continuously compute current users per geo-region KTable<UserId,  Location>  userLocations =  builder.table(“user-­‐locations-­‐topic”); KTable<UserId,  Prefs>        userPrefs =  builder.table(“user-­‐preferences-­‐topic”); //  Merge  into  detailed  user  profiles  (continuously  updated) KTable<UserId,  UserProfile>  userProfiles = userLocations.join(userPrefs,  (loc,  prefs)  -­‐>  new  UserProfile(loc,  prefs)); //  Compute  per-­‐region  statistics  (continuously  updated) KTable<UserId,  Long>  usersPerRegion =  userProfiles .filter((userId,  profile)    -­‐>  profile.age <  30) .groupBy((userId,  profile)  -­‐>  profile.location) .count(); alice Europe user-locations Africa 3 … … Asia 8 Europe 5 Africa 3 … … Asia 7 Europe 6 KTable usersPerRegion KTable usersPerRegion
108Confidential Motivating example: continuously compute current users per geo-region 4 7 5 3 2 8 4 7 6 3 2 7 Alice Real-time dashboard “How many users younger than 30y, per region?” alice Europe user-locations alice Asia, 25y, … bob Europe, 46y, … … … alice Europe, 25y, … bob Europe, 46y, … … … -1 +1 user-locations (mobile team) user-prefs (web team)
109Confidential Another common use case: continuous transformations • Example: to enrich an input stream (user clicks) with side data (current user profile) KStream alice /rental/p8454vb, 06:59 PM PDT user-clicks-topics (at 1M msgs/s) “facts”
110Confidential Another common use case: continuous transformations • Example: to enrich an input stream (user clicks) with side data (current user profile) KStream alice /rental/p8454vb, 06:59 PM PDT alice Asia, 25y bob Europe, 46y … … KTable user-profiles-topic user-clicks-topics (at 1M msgs/s) “facts” “dimensions”
111Confidential Another common use case: continuous transformations • Example: to enrich an input stream (user clicks) with side data (current user profile) KStream alice /rental/p8454vb, 06:59 PDT, Asia, 25y stream.JOIN(table) alice /rental/p8454vb, 06:59 PM PDT alice Asia, 25y bob Europe, 46y … … KTable user-profiles-topic user-clicks-topics (at 1M msgs/s) “facts” “dimensions”
112Confidential Another common use case: continuous transformations • Example: to enrich an input stream (user clicks) with side data (current user profile) KStream alice /rental/p8454vb, 06:59 PDT, Asia, 25y stream.JOIN(table) alice /rental/p8454vb, 06:59 PM PDT alice Asia, 25y bob Europe, 46y … … KTable alice Europe, 25y bob Europe, 46y … …alice Europe new update for alice from user-locations topic user-profiles-topic user-clicks-topics (at 1M msgs/s) “facts” “dimensions”
113Confidential Appendix: Interactive Queries A closer look
114Confidential Interactive Queries
115Confidential Interactive Queries charlie 3bob 5 alice 2
116Confidential Interactive Queries New  API  to  access local  state  stores  of an  app  instance charlie 3bob 5 alice 2
117Confidential Interactive Queries New  API  to  discover running  app  instances charlie 3bob 5 alice 2 “host1:4460” “host5:5307” “host3:4777”
118Confidential Interactive Queries You:  inter-­app  communication  (RPC  layer)

More Related Content

What's hot (20)

PPTX
Data Pipelines with Kafka Connect
Kaufman Ng
 
PPTX
Kafka presentation
Mohammed Fazuluddin
 
PDF
Apache Kafka Architecture & Fundamentals Explained
confluent
 
PPTX
Microservices in the Apache Kafka Ecosystem
confluent
 
PDF
Kafka 101 and Developer Best Practices
confluent
 
PDF
Introduction to Kafka Streams
Guozhang Wang
 
ODP
Stream processing using Kafka
Knoldus Inc.
 
PPTX
Spring Boot+Kafka: the New Enterprise Platform
VMware Tanzu
 
PDF
Apache kafka
NexThoughts Technologies
 
PPTX
A visual introduction to Apache Kafka
Paul Brebner
 
PPTX
Introduction to KSQL: Streaming SQL for Apache Kafka®
confluent
 
PDF
Apache kafka performance(throughput) - without data loss and guaranteeing dat...
SANG WON PARK
 
PPTX
Apache Kafka Best Practices
DataWorks Summit/Hadoop Summit
 
PDF
Integrating Apache Kafka Into Your Environment
confluent
 
PDF
Apache Arrow Flight: A New Gold Standard for Data Transport
Wes McKinney
 
PPTX
Apache Kafka
Saroj Panyasrivanit
 
PPTX
Getting up to speed with MirrorMaker 2 | Mickael Maison, IBM and Ryanne Dolan...
HostedbyConfluent
 
PPTX
Apache Kafka at LinkedIn
Guozhang Wang
 
PDF
Fundamentals of Apache Kafka
Chhavi Parasher
 
PPTX
Kafka MirrorMaker: Disaster Recovery, Scaling Reads, Isolate Mission Critical...
Jean-Paul Azar
 
Data Pipelines with Kafka Connect
Kaufman Ng
 
Kafka presentation
Mohammed Fazuluddin
 
Apache Kafka Architecture & Fundamentals Explained
confluent
 
Microservices in the Apache Kafka Ecosystem
confluent
 
Kafka 101 and Developer Best Practices
confluent
 
Introduction to Kafka Streams
Guozhang Wang
 
Stream processing using Kafka
Knoldus Inc.
 
Spring Boot+Kafka: the New Enterprise Platform
VMware Tanzu
 
Apache kafka
NexThoughts Technologies
 
A visual introduction to Apache Kafka
Paul Brebner
 
Introduction to KSQL: Streaming SQL for Apache Kafka®
confluent
 
Apache kafka performance(throughput) - without data loss and guaranteeing dat...
SANG WON PARK
 
Apache Kafka Best Practices
DataWorks Summit/Hadoop Summit
 
Integrating Apache Kafka Into Your Environment
confluent
 
Apache Arrow Flight: A New Gold Standard for Data Transport
Wes McKinney
 
Apache Kafka
Saroj Panyasrivanit
 
Getting up to speed with MirrorMaker 2 | Mickael Maison, IBM and Ryanne Dolan...
HostedbyConfluent
 
Apache Kafka at LinkedIn
Guozhang Wang
 
Fundamentals of Apache Kafka
Chhavi Parasher
 
Kafka MirrorMaker: Disaster Recovery, Scaling Reads, Isolate Mission Critical...
Jean-Paul Azar
 

Viewers also liked (20)

PDF
The Data Dichotomy- Rethinking the Way We Treat Data and Services
confluent
 
PDF
Apache kafka-a distributed streaming platform
confluent
 
PDF
Data integration with Apache Kafka
confluent
 
PPTX
Streaming in Practice - Putting Apache Kafka in Production
confluent
 
PPTX
Data Streaming with Apache Kafka & MongoDB
confluent
 
PPTX
Deep Dive into Apache Kafka
confluent
 
PDF
What's new in Confluent 3.2 and Apache Kafka 0.10.2
confluent
 
PDF
A Practical Guide to Selecting a Stream Processing Technology
confluent
 
PDF
Power of the Log: LSM & Append Only Data Structures
confluent
 
PDF
Demystifying Stream Processing with Apache Kafka
confluent
 
PPTX
AMIS SIG - Introducing Apache Kafka - Scalable, reliable Event Bus & Message ...
Lucas Jellema
 
PDF
Leveraging Mainframe Data for Modern Analytics
confluent
 
PDF
Monitoring Apache Kafka with Confluent Control Center
confluent
 
PPTX
Strata+Hadoop 2017 San Jose: Lessons from a year of supporting Apache Kafka
confluent
 
PDF
Strata+Hadoop 2017 San Jose - The Rise of Real Time: Apache Kafka and the Str...
confluent
 
PPTX
Introduction To Streaming Data and Stream Processing with Apache Kafka
confluent
 
PDF
Distributed stream processing with Apache Kafka
confluent
 
PDF
Data Pipelines Made Simple with Apache Kafka
confluent
 
PPTX
What is the Oracle PaaS Cloud for Developers (Oracle Cloud Day, The Netherlan...
Lucas Jellema
 
PPTX
Oracle OpenWorld 2016 Review - Focus on Data, BigData, Streaming Data, Machin...
Lucas Jellema
 
The Data Dichotomy- Rethinking the Way We Treat Data and Services
confluent
 
Apache kafka-a distributed streaming platform
confluent
 
Data integration with Apache Kafka
confluent
 
Streaming in Practice - Putting Apache Kafka in Production
confluent
 
Data Streaming with Apache Kafka & MongoDB
confluent
 
Deep Dive into Apache Kafka
confluent
 
What's new in Confluent 3.2 and Apache Kafka 0.10.2
confluent
 
A Practical Guide to Selecting a Stream Processing Technology
confluent
 
Power of the Log: LSM & Append Only Data Structures
confluent
 
Demystifying Stream Processing with Apache Kafka
confluent
 
AMIS SIG - Introducing Apache Kafka - Scalable, reliable Event Bus & Message ...
Lucas Jellema
 
Leveraging Mainframe Data for Modern Analytics
confluent
 
Monitoring Apache Kafka with Confluent Control Center
confluent
 
Strata+Hadoop 2017 San Jose: Lessons from a year of supporting Apache Kafka
confluent
 
Strata+Hadoop 2017 San Jose - The Rise of Real Time: Apache Kafka and the Str...
confluent
 
Introduction To Streaming Data and Stream Processing with Apache Kafka
confluent
 
Distributed stream processing with Apache Kafka
confluent
 
Data Pipelines Made Simple with Apache Kafka
confluent
 
What is the Oracle PaaS Cloud for Developers (Oracle Cloud Day, The Netherlan...
Lucas Jellema
 
Oracle OpenWorld 2016 Review - Focus on Data, BigData, Streaming Data, Machin...
Lucas Jellema
 
Ad

Similar to Introducing Kafka's Streams API (20)

PPTX
Kafka Streams: The Stream Processing Engine of Apache Kafka
Eno Thereska
 
PDF
Rethinking Stream Processing with Apache Kafka: Applications vs. Clusters, St...
Michael Noll
 
PPTX
Kafka Streams for Java enthusiasts
Slim Baltagi
 
PDF
A Tour of Apache Kafka
confluent
 
PPTX
Introducing Apache Kafka's Streams API - Kafka meetup Munich, Jan 25 2017
Michael Noll
 
PDF
Apache Kafka as Event Streaming Platform for Microservice Architectures
Kai Wähner
 
PPTX
Streaming Data and Stream Processing with Apache Kafka
confluent
 
PDF
Riviera Jug - 20/03/2018 - Kafka streams
Florent Ramiere
 
PDF
Apache Kafka as Event-Driven Open Source Streaming Platform (Prague Meetup)
Kai Wähner
 
PDF
How to Build Streaming Apps with Confluent II
confluent
 
PDF
Spark (Structured) Streaming vs. Kafka Streams - two stream processing platfo...
Guido Schmutz
 
PDF
Spark (Structured) Streaming vs. Kafka Streams
Guido Schmutz
 
PDF
Kafka Vienna Meetup 020719
Patrik Kleindl
 
PPTX
Building streaming data applications using Kafka*[Connect + Core + Streams] b...
Data Con LA
 
PDF
Confluent kafka meetupseattle jan2017
Nitin Kumar
 
PDF
Kafka Summit NYC 2017 Introduction to Kafka Streams with a Real-life Example
confluent
 
PDF
Kafka Summit SF 2017 - Query the Application, Not a Database: “Interactive Qu...
confluent
 
PDF
Apache Kafka - A Distributed Streaming Platform
Paolo Castagna
 
PDF
Introduction to Apache Kafka and why it matters - Madrid
Paolo Castagna
 
PDF
Kafka Connect and Streams (Concepts, Architecture, Features)
Kai Wähner
 
Kafka Streams: The Stream Processing Engine of Apache Kafka
Eno Thereska
 
Rethinking Stream Processing with Apache Kafka: Applications vs. Clusters, St...
Michael Noll
 
Kafka Streams for Java enthusiasts
Slim Baltagi
 
A Tour of Apache Kafka
confluent
 
Introducing Apache Kafka's Streams API - Kafka meetup Munich, Jan 25 2017
Michael Noll
 
Apache Kafka as Event Streaming Platform for Microservice Architectures
Kai Wähner
 
Streaming Data and Stream Processing with Apache Kafka
confluent
 
Riviera Jug - 20/03/2018 - Kafka streams
Florent Ramiere
 
Apache Kafka as Event-Driven Open Source Streaming Platform (Prague Meetup)
Kai Wähner
 
How to Build Streaming Apps with Confluent II
confluent
 
Spark (Structured) Streaming vs. Kafka Streams - two stream processing platfo...
Guido Schmutz
 
Spark (Structured) Streaming vs. Kafka Streams
Guido Schmutz
 
Kafka Vienna Meetup 020719
Patrik Kleindl
 
Building streaming data applications using Kafka*[Connect + Core + Streams] b...
Data Con LA
 
Confluent kafka meetupseattle jan2017
Nitin Kumar
 
Kafka Summit NYC 2017 Introduction to Kafka Streams with a Real-life Example
confluent
 
Kafka Summit SF 2017 - Query the Application, Not a Database: “Interactive Qu...
confluent
 
Apache Kafka - A Distributed Streaming Platform
Paolo Castagna
 
Introduction to Apache Kafka and why it matters - Madrid
Paolo Castagna
 
Kafka Connect and Streams (Concepts, Architecture, Features)
Kai Wähner
 
Ad

More from confluent (20)

PDF
Stream Processing Handson Workshop - Flink SQL Hands-on Workshop (Korean)
confluent
 
PPTX
Webinar Think Right - Shift Left - 19-03-2025.pptx
confluent
 
PDF
Migration, backup and restore made easy using Kannika
confluent
 
PDF
Five Things You Need to Know About Data Streaming in 2025
confluent
 
PDF
Data in Motion Tour Seoul 2024 - Keynote
confluent
 
PDF
Data in Motion Tour Seoul 2024 - Roadmap Demo
confluent
 
PDF
From Stream to Screen: Real-Time Data Streaming to Web Frontends with Conflue...
confluent
 
PDF
Confluent per il settore FSI: Accelerare l'Innovazione con il Data Streaming...
confluent
 
PDF
Data in Motion Tour 2024 Riyadh, Saudi Arabia
confluent
 
PDF
Build a Real-Time Decision Support Application for Financial Market Traders w...
confluent
 
PDF
Strumenti e Strategie di Stream Governance con Confluent Platform
confluent
 
PDF
Compose Gen-AI Apps With Real-Time Data - In Minutes, Not Weeks
confluent
 
PDF
Building Real-Time Gen AI Applications with SingleStore and Confluent
confluent
 
PDF
Unlocking value with event-driven architecture by Confluent
confluent
 
PDF
Il Data Streaming per un’AI real-time di nuova generazione
confluent
 
PDF
Unleashing the Future: Building a Scalable and Up-to-Date GenAI Chatbot with ...
confluent
 
PDF
Break data silos with real-time connectivity using Confluent Cloud Connectors
confluent
 
PDF
Building API data products on top of your real-time data infrastructure
confluent
 
PDF
Speed Wins: From Kafka to APIs in Minutes
confluent
 
PDF
Evolving Data Governance for the Real-time Streaming and AI Era
confluent
 
Stream Processing Handson Workshop - Flink SQL Hands-on Workshop (Korean)
confluent
 
Webinar Think Right - Shift Left - 19-03-2025.pptx
confluent
 
Migration, backup and restore made easy using Kannika
confluent
 
Five Things You Need to Know About Data Streaming in 2025
confluent
 
Data in Motion Tour Seoul 2024 - Keynote
confluent
 
Data in Motion Tour Seoul 2024 - Roadmap Demo
confluent
 
From Stream to Screen: Real-Time Data Streaming to Web Frontends with Conflue...
confluent
 
Confluent per il settore FSI: Accelerare l'Innovazione con il Data Streaming...
confluent
 
Data in Motion Tour 2024 Riyadh, Saudi Arabia
confluent
 
Build a Real-Time Decision Support Application for Financial Market Traders w...
confluent
 
Strumenti e Strategie di Stream Governance con Confluent Platform
confluent
 
Compose Gen-AI Apps With Real-Time Data - In Minutes, Not Weeks
confluent
 
Building Real-Time Gen AI Applications with SingleStore and Confluent
confluent
 
Unlocking value with event-driven architecture by Confluent
confluent
 
Il Data Streaming per un’AI real-time di nuova generazione
confluent
 
Unleashing the Future: Building a Scalable and Up-to-Date GenAI Chatbot with ...
confluent
 
Break data silos with real-time connectivity using Confluent Cloud Connectors
confluent
 
Building API data products on top of your real-time data infrastructure
confluent
 
Speed Wins: From Kafka to APIs in Minutes
confluent
 
Evolving Data Governance for the Real-time Streaming and AI Era
confluent
 

Recently uploaded (20)

PPTX
How Odoo Became a Game-Changer for an IT Company in Manufacturing ERP
SatishKumar2651
 
PPT
MergeSortfbsjbjsfk sdfik k
RafishaikIT02044
 
PDF
Continouous failure - Why do we make our lives hard?
Papp Krisztián
 
PPTX
Java Native Memory Leaks: The Hidden Villain Behind JVM Performance Issues
Tier1 app
 
PDF
Alexander Marshalov - How to use AI Assistants with your Monitoring system Q2...
VictoriaMetrics
 
PDF
Letasoft Sound Booster 1.12.0.538 Crack Download+ Product Key [Latest]
HyperPc soft
 
PDF
Unlock Efficiency with Insurance Policy Administration Systems
Insurance Tech Services
 
PDF
Mobile CMMS Solutions Empowering the Frontline Workforce
CryotosCMMSSoftware
 
PPTX
3uTools Full Crack Free Version Download [Latest] 2025
muhammadgurbazkhan
 
PDF
vMix Pro 28.0.0.42 Download vMix Registration key Bundle
kulindacore
 
PPTX
A Complete Guide to Salesforce SMS Integrations Build Scalable Messaging With...
360 SMS APP
 
PDF
Linux Certificate of Completion - LabEx Certificate
VICTOR MAESTRE RAMIREZ
 
PPTX
MailsDaddy Outlook OST to PST converter.pptx
abhishekdutt366
 
PDF
HiHelloHR – Simplify HR Operations for Modern Workplaces
HiHelloHR
 
PDF
Why Businesses Are Switching to Open Source Alternatives to Crystal Reports.pdf
Varsha Nayak
 
PDF
GetOnCRM Speeds Up Agentforce 3 Deployment for Enterprise AI Wins.pdf
GetOnCRM Solutions
 
PDF
Alarm in Android-Scheduling Timed Tasks Using AlarmManager in Android.pdf
Nabin Dhakal
 
PPTX
Feb 2021 Cohesity first pitch presentation.pptx
enginsayin1
 
PPTX
Hardware(Central Processing Unit ) CU and ALU
RizwanaKalsoom2
 
PPTX
Writing Better Code - Helping Developers make Decisions.pptx
Lorraine Steyn
 
How Odoo Became a Game-Changer for an IT Company in Manufacturing ERP
SatishKumar2651
 
MergeSortfbsjbjsfk sdfik k
RafishaikIT02044
 
Continouous failure - Why do we make our lives hard?
Papp Krisztián
 
Java Native Memory Leaks: The Hidden Villain Behind JVM Performance Issues
Tier1 app
 
Alexander Marshalov - How to use AI Assistants with your Monitoring system Q2...
VictoriaMetrics
 
Letasoft Sound Booster 1.12.0.538 Crack Download+ Product Key [Latest]
HyperPc soft
 
Unlock Efficiency with Insurance Policy Administration Systems
Insurance Tech Services
 
Mobile CMMS Solutions Empowering the Frontline Workforce
CryotosCMMSSoftware
 
3uTools Full Crack Free Version Download [Latest] 2025
muhammadgurbazkhan
 
vMix Pro 28.0.0.42 Download vMix Registration key Bundle
kulindacore
 
A Complete Guide to Salesforce SMS Integrations Build Scalable Messaging With...
360 SMS APP
 
Linux Certificate of Completion - LabEx Certificate
VICTOR MAESTRE RAMIREZ
 
MailsDaddy Outlook OST to PST converter.pptx
abhishekdutt366
 
HiHelloHR – Simplify HR Operations for Modern Workplaces
HiHelloHR
 
Why Businesses Are Switching to Open Source Alternatives to Crystal Reports.pdf
Varsha Nayak
 
GetOnCRM Speeds Up Agentforce 3 Deployment for Enterprise AI Wins.pdf
GetOnCRM Solutions
 
Alarm in Android-Scheduling Timed Tasks Using AlarmManager in Android.pdf
Nabin Dhakal
 
Feb 2021 Cohesity first pitch presentation.pptx
enginsayin1
 
Hardware(Central Processing Unit ) CU and ALU
RizwanaKalsoom2
 
Writing Better Code - Helping Developers make Decisions.pptx
Lorraine Steyn
 

Introducing Kafka's Streams API

  • 1. 1Confidential Introducing Kafka’s Streams API Stream processing made simple Target audience: technical staff, developers, architects Expected duration for full deck: 45 minutes
  • 2. 2Confidential 0.10 Data processing (Streams API) 0.9 Data integration (Connect API) Intra-cluster replication 0.8 Apache Kafka: birthed as a messaging system, now a streaming platform 2012 2014 2015 2016 2017 Cluster mirroring, data compression 0.7 2013
  • 3. 3Confidential Kafka’s Streams API: the easiest way to process data in Apache Kafka Key Benefits of Apache Kafka’s Streams API • Build Apps, Not Clusters: no additional cluster required • Cluster to go: elastic, scalable, distributed, fault-tolerant, secure • Database to go: tables, local state, interactive queries • Equally viable for S / M / L / XL / XXL use cases • “Runs Everywhere”: integrates with your existing deployment strategies such as containers, automation, cloud Part of open source Apache Kafka, introduced in 0.10+ • Powerful client library to build stream processing apps • Apps are standard Java applications that run on client machines • https://github.com/apache/kafka/tree/trunk/streams Streams API Your App Kafka Cluster
  • 4. 4Confidential Kafka’s Streams API: Unix analogy $  cat  <  in.txt | grep “apache”  | tr a-­‐z  A-­‐Z  > out.txt Kafka  Cluster Connect  API Streams  API
  • 5. 5Confidential Streams API in the context of Kafka Streams API Your App Kafka Cluster ConnectAPI ConnectAPI OtherSystems OtherSystems
  • 6. 6Confidential When to use Kafka’s Streams API • Mainstream Application Development • To build core business applications • Microservices • Fast Data apps for small and big data • Reactive applications • Continuous queries and transformations • Event-triggered processes • The “T” in ETL • <and more> Use case examples • Real-time monitoring and intelligence • Customer 360-degree view • Fraud detection • Location-based marketing • Fleet management • <and more>
  • 7. 7Confidential Some public use cases in the wild & external articles • Applying Kafka’s Streams API for internal message delivery pipeline at LINE Corp. • http://developers.linecorp.com/blog/?p=3960 • Kafka Streams in production at LINE, a social platform based in Japan with 220+ million users • Microservices and reactive applications at Capital One • https://speakerdeck.com/bobbycalderwood/commander-decoupled-immutable-rest-apis-with-kafka-streams • User behavior analysis • https://timothyrenner.github.io/engineering/2016/08/11/kafka-streams-not-looking-at-facebook.html • Containerized Kafka Streams applications in Scala • https://www.madewithtea.com/processing-tweets-with-kafka-streams.html • Geo-spatial data analysis • http://www.infolace.com/blog/2016/07/14/simple-spatial-windowing-with-kafka-streams/ • Language classification with machine learning • https://dzone.com/articles/machine-learning-with-kafka-streams
  • 9. 9Confidential Architecture comparison: use case example Real-time dashboard for security monitoring “Which of my data centers are under attack?”
  • 10. 10Confidential Architecture comparison: use case example Other App Dashboard Frontend App Other App 1 Capture business events in Kafka 2 Must process events with separate cluster (e.g. Spark) 4 Other apps access latest results by querying these DBs 3 Must share latest results through separate systems (e.g. MySQL) Before: Undue complexity, heavy footprint, many technologies, split ownership with conflicting priorities Your “Job” Other App Dashboard Frontend App Other App 1 Capture business events in Kafka 2 Process events with standard Java apps that use Kafka Streams 3 Now other apps can directly query the latest results With Kafka Streams: simplified, app-centric architecture, puts app owners in control Kafka Streams Your App
  • 13. 13Confidential How do I install the Streams API? • There is and there should be no “installation” – Build Apps, Not Clusters! • It’s a library. Add it to your app like any other library. <dependency> <groupId>org.apache.kafka</groupId> <artifactId>kafka-­‐streams</artifactId> <version>0.10.1.1</version> </dependency>
  • 14. 14Confidential “But wait a minute – where’s THE CLUSTER to process the data?” • No cluster needed – Build Apps, Not Clusters! • Unlearn bad habits: “do cool stuff with data ≠ must have cluster” Ok. Ok. Ok.
  • 15. 15Confidential Organizational benefits: decouple teams and roadmaps, scale people
  • 16. 16Confidential Organizational benefits: decouple teams and roadmaps, scale people Infrastructure Team (Kafka as a shared, multi-tenant service) Fraud detection app Payments team Recommenda tions app Mobile team Security alerts app Operations team ...more apps... ...
  • 17. 17Confidential How do I package, deploy, monitor my apps? How do I …? • Whatever works for you. Stick to what you/your company think is the best way. • No magic needed. • Why? Because an app that uses the Streams API is…a normal Java app.
  • 19. 19Confidential The API is but the tip of the iceberg API,  coding Org.  processes Reality™ Deployment Operations Security … Architecture Debugging
  • 20. 20Confidential • API option 1: DSL (declarative) KStream<Integer,  Integer>  input  = builder.stream("numbers-­‐topic"); //  Stateless  computation KStream<Integer,  Integer>  doubled  = input.mapValues(v  -­‐>  v  *  2); //  Stateful  computation KTable<Integer,  Integer>  sumOfOdds =  input .filter((k,v)  -­‐>  v  %  2  !=  0) .selectKey((k,  v)  -­‐>  1) .groupByKey() .reduce((v1,  v2)  -­‐>  v1  +  v2,  "sum-­‐of-­‐odds"); The preferred API for most use cases. Particularly appeals to: • Fans of Scala, functional programming • Users familiar with e.g. Spark
  • 21. 21Confidential • API option 2: Processor API (imperative) class  PrintToConsoleProcessor implements  Processor<K,  V>  { @Override public  void  init(ProcessorContext context)  {} @Override void  process(K  key,  V  value)  {   System.out.println("Got  value  "  +  value);   } @Override void  punctuate(long  timestamp)  {} @Override void  close()  {} } Full flexibility but more manual work Appeals to: • Users who require functionality that is not yet available in the DSL • Users familiar with e.g. Storm, Samza • Still, check out the DSL!
  • 22. 22Confidential When to use Kafka Streams vs. Kafka’s “normal” consumer clients Kafka Streams • Basically all the time • Basically all the time • Basically all the time • Basically all the time • Basically all the time • Basically all the time • Basically all the time • Basically all the time • Basically all the time • Basically all the time • Basically all the time Kafka consumer clients (Java, C/C++, Python, Go, …) • When you must interact with Kafka at a very low level and/or in a very special way • Example: When integrating your own stream processing tool (Spark, Storm) with Kafka.
  • 23. 23Confidential Code comparison Featuring Kafka with Streams API <-> Spark Streaming
  • 24. 24Confidential ”My WordCount is better than your WordCount” (?) Kafka Spark These isolated code snippets are nice (and actually quite similar) but they are not very meaningful. In practice, we also need to read data from somewhere, write data back to somewhere, etc.– but we can see none of this here.
  • 26. 26Confidential Compared to: WordCount in Spark 2.0 1 2 3 Runtime model leaks into processing logic (here: interfacing from Spark with Kafka)
  • 27. 27Confidential Compared to: WordCount in Spark 2.0 4 5 Runtime model leaks into processing logic (driver vs. executors)
  • 32. 32Confidential Streams and Tables Stream Processing meets Databases
  • 35. 35Confidential Key observation: close relationship between Streams and Tables http://www.confluent.io/blog/introducing-kafka-streams-stream-processing-made-simple http://docs.confluent.io/current/streams/concepts.html#duality-of-streams-and-tables
  • 37. 37Confidential Example: Streams and Tables in Kafka Word Count hello 2 kafka 1 world 1 … …
  • 42. 42Confidential Example: continuously compute current users per geo-region 4 7 5 3 2 8 4 7 6 3 2 7 Alice Real-time dashboard “How many users younger than 30y, per region?” alice Europe user-locations alice Asia, 25y, … bob Europe, 46y, … … … alice Europe, 25y, … bob Europe, 46y, … … … -1 +1 user-locations (mobile team) user-prefs (web team)
  • 43. 43Confidential Example: continuously compute current users per geo-region KTable<UserId,  Location>  userLocations =  builder.table(“user-­‐locations-­‐topic”); KTable<UserId,  Prefs>        userPrefs =  builder.table(“user-­‐preferences-­‐topic”);
  • 44. 44Confidential Example: continuously compute current users per geo-region alice Europe user-locations alice Asia, 25y, … bob Europe, 46y, … … … alice Europe, 25y, … bob Europe, 46y, … … … KTable<UserId,  Location>  userLocations =  builder.table(“user-­‐locations-­‐topic”); KTable<UserId,  Prefs>        userPrefs =  builder.table(“user-­‐preferences-­‐topic”); //  Merge  into  detailed  user  profiles  (continuously  updated) KTable<UserId,  UserProfile>  userProfiles = userLocations.join(userPrefs,  (loc,  prefs)  -­‐>  new  UserProfile(loc,  prefs)); KTable userProfilesKTable userProfiles
  • 45. 45Confidential Example: continuously compute current users per geo-region KTable<UserId,  Location>  userLocations =  builder.table(“user-­‐locations-­‐topic”); KTable<UserId,  Prefs>        userPrefs =  builder.table(“user-­‐preferences-­‐topic”); //  Merge  into  detailed  user  profiles  (continuously  updated) KTable<UserId,  UserProfile>  userProfiles = userLocations.join(userPrefs,  (loc,  prefs)  -­‐>  new  UserProfile(loc,  prefs)); //  Compute  per-­‐region  statistics  (continuously  updated) KTable<UserId,  Long>  usersPerRegion =  userProfiles .filter((userId,  profile)    -­‐>  profile.age <  30) .groupBy((userId,  profile)  -­‐>  profile.location) .count(); alice Europe user-locations Africa 3 … … Asia 8 Europe 5 Africa 3 … … Asia 7 Europe 6 KTable usersPerRegion KTable usersPerRegion
  • 46. 46Confidential Example: continuously compute current users per geo-region 4 7 5 3 2 8 4 7 6 3 2 7 Alice Real-time dashboard “How many users younger than 30y, per region?” alice Europe user-locations alice Asia, 25y, … bob Europe, 46y, … … … alice Europe, 25y, … bob Europe, 46y, … … … -1 +1 user-locations (mobile team) user-prefs (web team)
  • 48. 48Confidential Streams meet Tables • Most use cases for stream processing require both Streams and Tables • Essential for any stateful computations • Kafka ships with first-class support for Streams and Tables • Scalability, fault tolerance, efficient joins and aggregations, … • Benefits include: simplified architectures, less moving pieces, less Do-It-Yourself work
  • 50. 50Confidential Key features in 0.10 • Native, 100%-compatible Kafka integration
  • 51. 51Confidential Native, 100% compatible Kafka integration Read  from  Kafka Write  to  Kafka
  • 52. 52Confidential Key features in 0.10 • Native, 100%-compatible Kafka integration • Secure stream processing using Kafka’s security features
  • 53. 53Confidential Secure stream processing with the Streams API • Your applications can leverage all client-side security features in Apache Kafka • Security features include: • Encrypting data-in-transit between applications and Kafka clusters • Authenticating applications against Kafka clusters (“only some apps may talk to the production cluster”) • Authorizing application against Kafka clusters (“only some apps may read data from sensitive topics”)
  • 54. 54Confidential Configuring security settings • In general, you can configure both Kafka Streams plus the underlying Kafka clients in your apps
  • 55. 55Confidential Configuring security settings • Example: encrypting data-in-transit + client authentication to Kafka cluster Full demo application at https://github.com/confluentinc/examples
  • 56. 56Confidential Key features in 0.10 • Native, 100%-compatible Kafka integration • Secure stream processing using Kafka’s security features • Elastic and highly scalable • Fault-tolerant
  • 60. 60Confidential Key features in 0.10 • Native, 100%-compatible Kafka integration • Secure stream processing using Kafka’s security features • Elastic and highly scalable • Fault-tolerant • Stateful and stateless computations
  • 61. 61Confidential Stateful computations • Stateful computations like aggregations (e.g. counting), joins, or windowing require state • State stores are the backbone of state management • … are local for best performance • … are backed up to Kafka for elasticity and for fault-tolerance • ... are per stream task for isolation – think: share-nothing • Pluggable storage engines • Default: RocksDB (a key-value store) to allow for local state that is larger than available RAM • You can also use your own, custom storage engine • From the user perspective: • DSL: no need to worry about anything, state management is automatically being done for you • Processor API: direct access to state stores – very flexible but more manual work
  • 66. 66Confidential Use case: real-time, distributed joins at large scale
  • 67. 67Confidential Use case: real-time, distributed joins at large scale
  • 68. 68Confidential Use case: real-time, distributed joins at large scale
  • 69. 69Confidential Stateful computations • Use the Processor API to interact directly with state stores Get  the  store Use  the  store
  • 70. 70Confidential Key features in 0.10 • Native, 100%-compatible Kafka integration • Secure stream processing using Kafka’s security features • Elastic and highly scalable • Fault-tolerant • Stateful and stateless computations • Interactive queries
  • 72. 72Confidential Interactive Queries: architecture comparison Kafka Streams App App App App 1 Capture business events in Kafka 2 Process the events with Kafka Streams 4 Other apps query external systems for latest results ! Must use external systems to share latest results App App App 1 Capture business events in Kafka 2 Process the events with Kafka Streams 3 Now other apps can directly query the latest results Before (0.10.0) After (0.10.1): simplified, more app-centric architecture Kafka Streams App
  • 73. 73Confidential Key features in 0.10 • Native, 100%-compatible Kafka integration • Secure stream processing using Kafka’s security features • Elastic and highly scalable • Fault-tolerant • Stateful and stateless computations • Interactive queries • Time model
  • 76. 76Confidential Time • You configure the desired time semantics through timestamp extractors • Default extractor yields event-time semantics • Extracts embedded timestamps of Kafka messages (introduced in v0.10)
  • 77. 77Confidential Key features in 0.10 • Native, 100%-compatible Kafka integration • Secure stream processing using Kafka’s security features • Elastic and highly scalable • Fault-tolerant • Stateful and stateless computations • Interactive queries • Time model • Windowing
  • 78. 78Confidential Windowing • Group events in a stream using time-based windows • Use case examples: • Time-based analysis of ad impressions (”number of ads clicked in the past hour”) • Monitoring statistics of telemetry data (“1min/5min/15min averages”) Input data, where colors represent different users events Rectangles denote different event-time windows processing-time event-time windowing alice bob dave
  • 79. 79Confidential Windowing in the DSL TimeWindows.of(3000) TimeWindows.of(3000).advanceBy(1000)
  • 80. 80Confidential Key features in 0.10 • Native, 100%-compatible Kafka integration • Secure stream processing using Kafka’s security features • Elastic and highly scalable • Fault-tolerant • Stateful and stateless computations • Interactive queries • Time model • Windowing • Supports late-arriving and out-of-order data
  • 81. 81Confidential Out-of-order and late-arriving data • Is very common in practice, not a rare corner case • Related to time model discussion
  • 82. 82Confidential Out-of-order and late-arriving data: example when this will happen Users with mobile phones enter airplane, lose Internet connectivity Emails are being written during the 10h flight Internet connectivity is restored, phones will send queued emails now
  • 83. 83Confidential Out-of-order and late-arriving data • Is very common in practice, not a rare corner case • Related to time model discussion • We want control over how out-of-order data is handled, and handling must be efficient • Example: We process data in 5-minute windows, e.g. compute statistics • Option A: When event arrives 1 minute late: update the original result! • Option B: When event arrives 2 hours late: discard it!
  • 84. 84Confidential Key features in 0.10 • Native, 100%-compatible Kafka integration • Secure stream processing using Kafka’s security features • Elastic and highly scalable • Fault-tolerant • Stateful and stateless computations • Interactive queries • Time model • Windowing • Supports late-arriving and out-of-order data • Millisecond processing latency, no micro-batching • At-least-once processing guarantees (exactly-once is in the works as we speak)
  • 86. 86Confidential Roadmap outlook for Kafka Streams • Exactly-Once processing semantics • Unified API for real-time processing and “batch” processing • Global KTables • Session windows • … and more …
  • 88. 88Confidential Where to go from here • Kafka Streams is available in Confluent Platform 3.1 and in Apache Kafka 0.10.1 • http://www.confluent.io/download • Kafka Streams demos: https://github.com/confluentinc/examples • Java 7, Java 8+ with lambdas, and Scala • WordCount, Interactive Queries, Joins, Security, Windowing, Avro integration, … • Confluent documentation: http://docs.confluent.io/current/streams/ • Quickstart, Concepts, Architecture, Developer Guide, FAQ • Recorded talks • Introduction to Kafka Streams: http://www.youtube.com/watch?v=o7zSLNiTZbA • Application Development and Data in the Emerging World of Stream Processing (higher level talk): https://www.youtube.com/watch?v=JQnNHO5506w
  • 90. 90Confidential Appendix: Streams and Tables A closer look
  • 91. 91Confidential Motivating example: continuously compute current users per geo-region 4 7 5 3 2 8 Real-time dashboard “How many users younger than 30y, per region?” alice Asia, 25y, … bob Europe, 46y, … … … user-locations (mobile team) user-prefs (web team)
  • 92. 92Confidential Motivating example: continuously compute current users per geo-region 4 7 5 3 2 8 Real-time dashboard “How many users younger than 30y, per region?” alice Europe user-locations alice Asia, 25y, … bob Europe, 46y, … … … user-locations (mobile team) user-prefs (web team)
  • 93. 93Confidential Motivating example: continuously compute current users per geo-region 4 7 5 3 2 8 Real-time dashboard “How many users younger than 30y, per region?” alice Europe user-locations user-locations (mobile team) user-prefs (web team) alice Asia, 25y, … bob Europe, 46y, … … … alice Europe, 25y, … bob Europe, 46y, … … …
  • 94. 94Confidential Motivating example: continuously compute current users per geo-region 4 7 5 3 2 8 4 7 6 3 2 7 Alice Real-time dashboard “How many users younger than 30y, per region?” alice Europe user-locations alice Asia, 25y, … bob Europe, 46y, … … … alice Europe, 25y, … bob Europe, 46y, … … … -1 +1 user-locations (mobile team) user-prefs (web team)
  • 95. 95Confidential Same data, but different use cases require different interpretations alice San Francisco alice New York City alice Rio de Janeiro alice Sydney alice Beijing alice Paris alice Berlin
  • 96. 96Confidential Same data, but different use cases require different interpretations alice San Francisco alice New York City alice Rio de Janeiro alice Sydney alice Beijing alice Paris alice Berlin Use  case  1:  Frequent  traveler  status? Use  case  2:  Current  location?
  • 97. 97Confidential Same data, but different use cases require different interpretations “Alice has been to SFO, NYC, Rio, Sydney, Beijing, Paris, and finally Berlin.” “Alice is in SFO, NYC, Rio, Sydney, Beijing, Paris, Berlin right now.” ⚑ ⚑ ⚑⚑ ⚑ ⚑ ⚑ ⚑ ⚑ ⚑⚑ ⚑ ⚑ ⚑ Use  case  1:  Frequent  traveler  status? Use  case  2:  Current  location?
  • 98. 98Confidential Same data, but different use cases require different interpretations alice San Francisco alice New York City alice Rio de Janeiro alice Sydney alice Beijing alice Paris alice Berlin Use  case  1:  Frequent  traveler  status? Use  case  2:  Current  location? ⚑ ⚑ ⚑⚑ ⚑ ⚑⚑ ⚑
  • 99. 99Confidential Same data, but different use cases require different interpretations alice San Francisco alice New York City alice Rio de Janeiro alice Sydney alice Beijing alice Paris alice Berlin Use  case  1:  Frequent  traveler  status? Use  case  2:  Current  location? ⚑ ⚑ ⚑⚑ ⚑ ⚑⚑ ⚑
  • 100. 100Confidential Same data, but different use cases require different interpretations alice San Francisco alice New York City alice Rio de Janeiro alice Sydney alice Beijing alice Paris alice Berlin Use  case  1:  Frequent  traveler  status? Use  case  2:  Current  location? ⚑ ⚑ ⚑⚑ ⚑ ⚑⚑ ⚑
  • 101. 101Confidential Streams meet Tables record stream When you need… so that the topic is interpreted as a All the values of a key KStream then you’d read the Kafka topic into a Example All the places Alice has ever been to with messages interpreted as INSERT (append)
  • 102. 102Confidential Streams meet Tables record stream changelog stream When you need… so that the topic is interpreted as a All the values of a key Latest value of a key KStream KTable then you’d read the Kafka topic into a Example All the places Alice has ever been to Where Alice is right now with messages interpreted as INSERT (append) UPSERT (overwrite existing)
  • 103. 103Confidential Same data, but different use cases require different interpretations “Alice has been to SFO, NYC, Rio, Sydney, Beijing, Paris, and finally Berlin.” “Alice is in SFO, NYC, Rio, Sydney, Beijing, Paris, Berlin right now.” ⚑ ⚑ ⚑⚑ ⚑ ⚑ ⚑ ⚑ ⚑ ⚑⚑ ⚑ ⚑ ⚑ Use  case  1:  Frequent  traveler  status? Use  case  2:  Current  location? KStream KTable
  • 104. 104Confidential Motivating example: continuously compute current users per geo-region 4 7 5 3 2 8 4 7 6 3 2 7 Alice Real-time dashboard “How many users younger than 30y, per region?” alice Europe user-locations alice Asia, 25y, … bob Europe, 46y, … … … alice Europe, 25y, … bob Europe, 46y, … … … -1 +1 user-locations (mobile team) user-prefs (web team)
  • 105. 105Confidential Motivating example: continuously compute current users per geo-region KTable<UserId,  Location>  userLocations =  builder.table(“user-­‐locations-­‐topic”); KTable<UserId,  Prefs>        userPrefs =  builder.table(“user-­‐preferences-­‐topic”);
  • 106. 106Confidential Motivating example: continuously compute current users per geo-region alice Europe user-locations alice Asia, 25y, … bob Europe, 46y, … … … alice Europe, 25y, … bob Europe, 46y, … … … KTable<UserId,  Location>  userLocations =  builder.table(“user-­‐locations-­‐topic”); KTable<UserId,  Prefs>        userPrefs =  builder.table(“user-­‐preferences-­‐topic”); //  Merge  into  detailed  user  profiles  (continuously  updated) KTable<UserId,  UserProfile>  userProfiles = userLocations.join(userPrefs,  (loc,  prefs)  -­‐>  new  UserProfile(loc,  prefs)); KTable userProfilesKTable userProfiles
  • 107. 107Confidential Motivating example: continuously compute current users per geo-region KTable<UserId,  Location>  userLocations =  builder.table(“user-­‐locations-­‐topic”); KTable<UserId,  Prefs>        userPrefs =  builder.table(“user-­‐preferences-­‐topic”); //  Merge  into  detailed  user  profiles  (continuously  updated) KTable<UserId,  UserProfile>  userProfiles = userLocations.join(userPrefs,  (loc,  prefs)  -­‐>  new  UserProfile(loc,  prefs)); //  Compute  per-­‐region  statistics  (continuously  updated) KTable<UserId,  Long>  usersPerRegion =  userProfiles .filter((userId,  profile)    -­‐>  profile.age <  30) .groupBy((userId,  profile)  -­‐>  profile.location) .count(); alice Europe user-locations Africa 3 … … Asia 8 Europe 5 Africa 3 … … Asia 7 Europe 6 KTable usersPerRegion KTable usersPerRegion
  • 108. 108Confidential Motivating example: continuously compute current users per geo-region 4 7 5 3 2 8 4 7 6 3 2 7 Alice Real-time dashboard “How many users younger than 30y, per region?” alice Europe user-locations alice Asia, 25y, … bob Europe, 46y, … … … alice Europe, 25y, … bob Europe, 46y, … … … -1 +1 user-locations (mobile team) user-prefs (web team)
  • 109. 109Confidential Another common use case: continuous transformations • Example: to enrich an input stream (user clicks) with side data (current user profile) KStream alice /rental/p8454vb, 06:59 PM PDT user-clicks-topics (at 1M msgs/s) “facts”
  • 110. 110Confidential Another common use case: continuous transformations • Example: to enrich an input stream (user clicks) with side data (current user profile) KStream alice /rental/p8454vb, 06:59 PM PDT alice Asia, 25y bob Europe, 46y … … KTable user-profiles-topic user-clicks-topics (at 1M msgs/s) “facts” “dimensions”
  • 111. 111Confidential Another common use case: continuous transformations • Example: to enrich an input stream (user clicks) with side data (current user profile) KStream alice /rental/p8454vb, 06:59 PDT, Asia, 25y stream.JOIN(table) alice /rental/p8454vb, 06:59 PM PDT alice Asia, 25y bob Europe, 46y … … KTable user-profiles-topic user-clicks-topics (at 1M msgs/s) “facts” “dimensions”
  • 112. 112Confidential Another common use case: continuous transformations • Example: to enrich an input stream (user clicks) with side data (current user profile) KStream alice /rental/p8454vb, 06:59 PDT, Asia, 25y stream.JOIN(table) alice /rental/p8454vb, 06:59 PM PDT alice Asia, 25y bob Europe, 46y … … KTable alice Europe, 25y bob Europe, 46y … …alice Europe new update for alice from user-locations topic user-profiles-topic user-clicks-topics (at 1M msgs/s) “facts” “dimensions”
  • 116. 116Confidential Interactive Queries New  API  to  access local  state  stores  of an  app  instance charlie 3bob 5 alice 2
  • 117. 117Confidential Interactive Queries New  API  to  discover running  app  instances charlie 3bob 5 alice 2 “host1:4460” “host5:5307” “host3:4777”
  • 118. 118Confidential Interactive Queries You:  inter-­app  communication  (RPC  layer)