SlideShare a Scribd company logo

Introduction to Kafka Streams

Guozhang Wang
Guozhang Wang

The document provides an overview of stream processing using Kafka, outlining its key concepts, architecture, and the advantages of using Kafka for real-time data streaming. It discusses the Kafka Streams library introduced in Apache Kafka since version 0.10, highlighting features such as event-at-a-time processing, state management, and fault tolerance. The document also includes examples of Kafka Streams DSL and explains the challenges of stream processing such as ordering, partitioning, and state management.

Engineering
1 of 136
Downloaded 1,529 times
1
2
3
4
5
Most read
6
Most read
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
Most read
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
84
85
86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
Kafka Streams Stream processing Made Simple with Kafka 1 Guozhang Wang Hadoop Summit, June 28, 2016
2 What is NOT Stream Processing?
3 Stream Processing isn’t (necessarily) • Transient, approximate, lossy… • .. that you must have batch processing as safety net
4
5
6
7
8 Stream Processing • A different programming paradigm • .. that brings computation to unbounded data • .. with tradeoffs between latency / cost / correctness
9 Why Kafka in Stream Processing?
10 • Persistent Buffering • Logical Ordering • Scalable “source-of-truth” Kafka: Real-time Platforms
11 Stream Processing with Kafka
12 • Option I: Do It Yourself ! Stream Processing with Kafka
13 • Option I: Do It Yourself ! Stream Processing with Kafka while (isRunning) { // read some messages from Kafka inputMessages = consumer.poll(); // do some processing… // send output messages back to Kafka producer.send(outputMessages); }
14
15 • Ordering • Partitioning &

 Scalability
 • Fault tolerance DIY Stream Processing is Hard • State Management • Time, Window &

 Out-of-order Data
 • Re-processing
16 • Option I: Do It Yourself ! • Option II: full-fledged stream processing system • Storm, Spark, Flink, Samza, .. Stream Processing with Kafka
17 MapReduce Heritage? • Config Management • Resource Management
 • Configuration
 • etc..
18 MapReduce Heritage? • Config Management • Resource Management
 • Deployment
 • etc..
19 MapReduce Heritage? • Config Management • Resource Management
 • Deployment
 • etc.. Can I just use my own?!
20 • Option I: Do It Yourself ! • Option II: full-fledged stream processing system • Option III: lightweight stream processing library Stream Processing with Kafka
Kafka Streams • In Apache Kafka since v0.10, May 2016 • Powerful yet easy-to-use stream processing library • Event-at-a-time, Stateful • Windowing with out-of-order handling • Highly scalable, distributed, fault tolerant • and more.. 21
22 Anywhere, anytime Ok. Ok. Ok. Ok.
23 Anywhere, anytime <dependency>
 <groupId>org.apache.kafka</groupId> <artifactId>kafka-streams</artifactId> <version>0.10.0.0</version> </dependency>
24 Anywhere, anytime War File Rsync Puppet/Chef YARN M esos Docker Kubernetes Very Uncool Very Cool
25 Simple is Beautiful
Kafka Streams DSL 26 public static void main(String[] args) { // specify the processing topology by first reading in a stream from a topic KStream<String, String> words = builder.stream(”topic1”); // count the words in this stream as an aggregated table KTable<String, Long> counts = words.countByKey(”Counts”); // write the result table to a new topic counts.to(”topic2”); // create a stream processing instance and start running it KafkaStreams streams = new KafkaStreams(builder, config); streams.start(); }
Kafka Streams DSL 27 public static void main(String[] args) { // specify the processing topology by first reading in a stream from a topic KStream<String, String> words = builder.stream(”topic1”); // count the words in this stream as an aggregated table KTable<String, Long> counts = words.countByKey(”Counts”); // write the result table to a new topic counts.to(”topic2”); // create a stream processing instance and start running it KafkaStreams streams = new KafkaStreams(builder, config); streams.start(); }
Kafka Streams DSL 28 public static void main(String[] args) { // specify the processing topology by first reading in a stream from a topic KStream<String, String> words = builder.stream(”topic1”); // count the words in this stream as an aggregated table KTable<String, Long> counts = words.countByKey(”Counts”); // write the result table to a new topic counts.to(”topic2”); // create a stream processing instance and start running it KafkaStreams streams = new KafkaStreams(builder, config); streams.start(); }
Kafka Streams DSL 29 public static void main(String[] args) { // specify the processing topology by first reading in a stream from a topic KStream<String, String> words = builder.stream(”topic1”); // count the words in this stream as an aggregated table KTable<String, Long> counts = words.countByKey(”Counts”); // write the result table to a new topic counts.to(”topic2”); // create a stream processing instance and start running it KafkaStreams streams = new KafkaStreams(builder, config); streams.start(); }
Kafka Streams DSL 30 public static void main(String[] args) { // specify the processing topology by first reading in a stream from a topic KStream<String, String> words = builder.stream(”topic1”); // count the words in this stream as an aggregated table KTable<String, Long> counts = words.countByKey(”Counts”); // write the result table to a new topic counts.to(”topic2”); // create a stream processing instance and start running it KafkaStreams streams = new KafkaStreams(builder, config); streams.start(); }
Kafka Streams DSL 31 public static void main(String[] args) { // specify the processing topology by first reading in a stream from a topic KStream<String, String> words = builder.stream(”topic1”); // count the words in this stream as an aggregated table KTable<String, Long> counts = words.countByKey(”Counts”); // write the result table to a new topic counts.to(”topic2”); // create a stream processing instance and start running it KafkaStreams streams = new KafkaStreams(builder, config); streams.start(); }
32 Native Kafka Integration Property cfg = new Properties(); cfg.put(StreamsConfig.APPLICATION_ID_CONFIG, “my-streams-app”); cfg.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, “broker1:9092”); cfg.put(ConsumerConfig.AUTO_OFFSET_RESET_CONIFG, “earliest”); cfg.put(CommonClientConfigs.SECURITY_PROTOCOL_CONFIG, “SASL_SSL”); cfg.put(KafkaAvroSerDeConfig.SCHEMA_REGISTRY_URL_CONFIG, “registry:8081”); StreamsConfig config = new StreamsConfig(cfg); … KafkaStreams streams = new KafkaStreams(builder, config);
33 Property cfg = new Properties(); cfg.put(StreamsConfig.APPLICATION_ID_CONFIG, “my-streams-app”); cfg.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, “broker1:9092”); cfg.put(ConsumerConfig.AUTO_OFFSET_RESET_CONIFG, “earliest”); cfg.put(CommonClientConfigs.SECURITY_PROTOCOL_CONFIG, “SASL_SSL”); cfg.put(KafkaAvroSerDeConfig.SCHEMA_REGISTRY_URL_CONFIG, “registry:8081”); StreamsConfig config = new StreamsConfig(cfg); … KafkaStreams streams = new KafkaStreams(builder, config); Native Kafka Integration
34 API, coding “Full stack” evaluation Operations, debugging, …
35 API, coding “Full stack” evaluation Operations, debugging, … Simple is Beautiful
36 Key Idea: Outsource hard problems to Kafka!
Kafka Concepts: the Log 4 5 5 7 8 9 10 11 12... Producer Write Consumer1 Reads (offset 7) Consumer2 Reads (offset 10) Messages 3
Topic 1 Topic 2 Partitions Producers Producers Consumers Consumers Brokers Kafka Concepts: the Log
39 Kafka Streams: Key Concepts
Stream and Records 40 Key Value Key Value Key Value Key Value Stream Record
Processor Topology 41 Stream
Processor Topology 42 Stream Processor
Processor Topology 43 KStream<..> stream1 = builder.stream(”topic1”); KStream<..> stream2 = builder.stream(”topic2”); KStream<..> joined = stream1.leftJoin(stream2, ...); KTable<..> aggregated = joined.aggregateByKey(...); aggregated.to(”topic3”);
Processor Topology 44 KStream<..> stream1 = builder.stream(”topic1”); KStream<..> stream2 = builder.stream(”topic2”); KStream<..> joined = stream1.leftJoin(stream2, ...); KTable<..> aggregated = joined.aggregateByKey(...); aggregated.to(”topic3”);
Processor Topology 45 KStream<..> stream1 = builder.stream(”topic1”); KStream<..> stream2 = builder.stream(”topic2”); KStream<..> joined = stream1.leftJoin(stream2, ...); KTable<..> aggregated = joined.aggregateByKey(...); aggregated.to(”topic3”);
Processor Topology 46 KStream<..> stream1 = builder.stream(”topic1”); KStream<..> stream2 = builder.stream(”topic2”); KStream<..> joined = stream1.leftJoin(stream2, ...); KTable<..> aggregated = joined.aggregateByKey(...); aggregated.to(”topic3”);
Processor Topology 47 KStream<..> stream1 = builder.stream(”topic1”); KStream<..> stream2 = builder.stream(”topic2”); KStream<..> joined = stream1.leftJoin(stream2, ...); KTable<..> aggregated = joined.aggregateByKey(...); aggregated.to(”topic3”);
Processor Topology 48 Source Processor Sink Processor KStream<..> stream1 = builder.stream( KStream<..> stream2 = builder.stream( aggregated.to(
Processor Topology 49 KStream<..> stream1 = builder.stream(”topic1”); KStream<..> stream2 = builder.table(”topic2”); KStream<..> joined = stream1.leftJoin(stream2, ...); KTable<..> aggregated = joined.aggregateByKey(...); aggregated.to(”topic3”); builder.addSource(”Source1”, ”topic1”) .addSource(”Source2”, ”topic2”) .addProcessor(”Join”, MyJoin:new, ”Source1”, ”Source2”) .addProcessor(”Aggregate”, MyAggregate:new, ”Join”) .addStateStore(Stores.persistent().build(), ”Aggregate”) .addSink(”Sink”, ”topic3”, ”Aggregate”)
Processor Topology 50 builder.addSource(”Source1”, ”topic1”) .addSource(”Source2”, ”topic2”) .addProcessor(”Join”, MyJoin:new, ”Source1”, ”Source2”) .addProcessor(”Aggregate”, MyAggregate:new, ”Join”) .addStateStore(Stores.persistent().build(), ”Aggregate”) .addSink(”Sink”, ”topic3”, ”Aggregate”)
Processor Topology 51 builder.addSource(”Source1”, ”topic1”) .addSource(”Source2”, ”topic2”) .addProcessor(”Join”, MyJoin:new, ”Source1”, ”Source2”) .addProcessor(”Aggregate”, MyAggregate:new, ”Join”) .addStateStore(Stores.persistent().build(), ”Aggregate”) .addSink(”Sink”, ”topic3”, ”Aggregate”)
Processor Topology 52 builder.addSource(”Source1”, ”topic1”) .addSource(”Source2”, ”topic2”) .addProcessor(”Join”, MyJoin:new, ”Source1”, ”Source2”) .addProcessor(”Aggregate”, MyAggregate:new, ”Join”) .addStateStore(Stores.persistent().build(), ”Aggregate”) .addSink(”Sink”, ”topic3”, ”Aggregate”)
Processor Topology 53Kafka Streams Kafka
Processor Topology 54 … sink1.to(”topic1”); source1 = builder.table(”topic1”); source2 = sink1.through(”topic2”); …
Processor Topology 55 … sink1.to(”topic1”); source1 = builder.table(”topic1”); source2 = sink1.through(”topic2”); …
Processor Topology 56 … sink1.to(”topic1”); source1 = builder.table(”topic1”); source2 = sink1.through(”topic2”); …
Processor Topology 57 … sink1.to(”topic1”); source1 = builder.table(”topic1”); source2 = sink1.through(”topic2”); …
Processor Topology 58 … sink1.to(”topic1”); source1 = builder.table(”topic1”); source2 = sink1.through(”topic2”); … Sub-Topology
Processor Topology 59Kafka Streams Kafka
Processor Topology 60Kafka Streams Kafka
Processor Topology 61Kafka Streams Kafka
Processor Topology 62Kafka Streams Kafka
Stream Partitions and Tasks 63 Kafka Topic B Kafka Topic A P1 P2 P1 P2
Stream Partitions and Tasks 64 Kafka Topic B Kafka Topic A Processor Topology P1 P2 P1 P2
Stream Partitions and Tasks 65 Kafka Topic AKafka Topic B
Kafka Topic B Task2Task1 Stream Partitions and Tasks 66 Kafka Topic A
Kafka Topic B Stream Partitions and Tasks 67 Kafka Topic A Task2Task1
Kafka Topic B Stream Threads 68 Kafka Topic A MyApp.1 Task2Task1
Kafka Topic B Stream Threads 69 Kafka Topic A Task2Task1 MyApp.1 MyApp.2
Kafka Topic B Stream Threads 70 Kafka Topic A MyApp.1 MyApp.2 Task2Task1
Stream Threads 71 Kafka Topic AKafka Topic B Task2Task1 MyApp.1 MyApp.2
Stream Threads 72 Task3 MyApp.3 Kafka Topic AKafka Topic B Task2Task1 MyApp.1 MyApp.2
Stream Threads 73 Task3 Kafka Topic AKafka Topic B Task2Task1 MyApp.1 MyApp.2 MyApp.3
Stream Threads 74 Thread1 Kafka Topic B Task2Task1 Thread2 Task4Task3 Kafka Topic AKafka Topic A
Stream Threads 75 Thread1 Kafka Topic B Task2Task1 Thread2 Task4Task3 Kafka Topic AKafka Topic A
Stream Threads 76 Thread1 Kafka Topic B Task2Task1 Thread2 Task4Task3 Kafka Topic AKafka Topic A
Stream Threads 77 Thread1 Kafka Topic B Task2Task1 Thread2 Task4Task3 Kafka Topic AKafka Topic A
78 • Ordering • Partitioning &

 Scalability
 • Fault tolerance Stream Processing Hard Parts • State Management • Time, Window &

 Out-of-order Data
 • Re-processing
States in Stream Processing 79 • filter • map
 • join
 • aggregate Stateless Stateful
80
States in Stream Processing 81 KStream<..> stream1 = builder.stream(”topic1”); KStream<..> stream2 = builder.stream(”topic2”); KStream<..> joined = stream1.leftJoin(stream2, ...); KTable<..> aggregated = joined.aggregateByKey(...); aggregated.to(”topic2”); State
82 builder.addSource(”Source1”, ”topic1”) .addSource(”Source2”, ”topic2”) .addProcessor(”Join”, MyJoin:new, ”Source1”, ”Source2”) .addProcessor(”Aggregate”, MyAggregate:new, ”Join”) .addStateStore(Stores.persistent().build(), ”Aggregate”) .addSink(”Sink”, ”topic3”, ”Aggregate”) State States in Stream Processing
Kafka Topic B Task2Task1 States in Stream Processing 83 Kafka Topic A State State
It’s all about Time • Event-time (when an event is created) • Processing-time (when an event is processed) 84
Event-time 1 2 3 4 5 6 7 Processing-time 1999 2002 2005 1997 1980 1983 2015 85 PHANTOMMENACE ATTACKOFTHECLONES REVENGEOFTHESITH ANEWHOPE THEEMPIRESTRIKESBACK RETURNOFTHEJEDI THEFORCEAWAKENS Out-of-Order
Timestamp Extractor 86 public long extract(ConsumerRecord<Object, Object> record) { return System.currentTimeMillis(); } public long extract(ConsumerRecord<Object, Object> record) { return record.timestamp(); }
Timestamp Extractor 87 public long extract(ConsumerRecord<Object, Object> record) { return System.currentTimeMillis(); } public long extract(ConsumerRecord<Object, Object> record) { return record.timestamp(); } processing-time
Timestamp Extractor 88 public long extract(ConsumerRecord<Object, Object> record) { return System.currentTimeMillis(); } public long extract(ConsumerRecord<Object, Object> record) { return record.timestamp(); } processing-time event-time
Timestamp Extractor 89 public long extract(ConsumerRecord<Object, Object> record) { return System.currentTimeMillis(); } processing-time event-time public long extract(ConsumerRecord<Object, Object> record) { return ((JsonNode) record.value()).get(”timestamp”).longValue(); }
Windowing 90 t …
Windowing 91 t …
Windowing 92 t …
Windowing 93 t …
Windowing 94 t …
Windowing 95 t …
Windowing 96 t …
97 • Ordering • Partitioning &

 Scalability
 • Fault tolerance Stream Processing Hard Parts • State Management • Time, Window &

 Out-of-order Data
 • Re-processing
Stream v.s.Table? 98 KStream<..> stream1 = builder.stream(”topic1”); KStream<..> stream2 = builder.stream(”topic2”); KStream<..> joined = stream1.leftJoin(stream2, ...); KTable<..> aggregated = joined.aggregateByKey(...); aggregated.to(”topic2”); State
99 Tables ≈ Streams
100
101
102
The Stream-Table Duality • A stream is a changelog of a table • A table is a materialized view at time of a stream • Example: change data capture (CDC) of databases 103
KStream = interprets data as record stream ~ think: “append-only” KTable = data as changelog stream ~ continuously updated materialized view 104
105 alice eggs bob lettuce alice milk alice lnkd bob googl alice msft KStream KTable User purchase history User employment profile
106 alice eggs bob lettuce alice milk alice lnkd bob googl alice msft KStream KTable User purchase history User employment profile time “Alice bought eggs.” “Alice is now at LinkedIn.”
107 alice eggs bob lettuce alice milk alice lnkd bob googl alice msft KStream KTable User purchase history User employment profile time “Alice bought eggs and milk.” “Alice is now at LinkedIn Microsoft.”
108 alice 2 bob 10 alice 3 timeKStream.aggregate() KTable.aggregate() (key: Alice, value: 2) (key: Alice, value: 2)
109 alice 2 bob 10 alice 3 time (key: Alice, value: 2 3) (key: Alice, value: 2+3) KStream.aggregate() KTable.aggregate()
110 KStream KTable reduce() aggregate() … toStream() map() filter() join() … map() filter() join() …
111 KTable aggregated KStream joined KStream stream1KStream stream2 Updates Propagation in KTable State
112 KTable aggregated KStream joined KStream stream1KStream stream2 State Updates Propagation in KTable
113 KTable aggregated KStream joined KStream stream1KStream stream2 State Updates Propagation in KTable
114 KTable aggregated KStream joined KStream stream1KStream stream2 State Updates Propagation in KTable
115 • Ordering • Partitioning &

 Scalability
 • Fault tolerance Stream Processing Hard Parts • State Management • Time, Window &

 Out-of-order Data
 • Re-processing
116 Remember?
117 StateProcess StateProcess StateProcess Kafka ChangelogFault Tolerance Kafka Kafka Streams Kafka
118 StateProcess StateProcess Protoco l StateProcess Fault Tolerance Kafka Kafka Streams Kafka Changelog Kafka
119 StateProcess StateProcess Protoco l StateProcess Fault Tolerance StateProcess Kafka Kafka Streams Kafka Changelog Kafka
120
121
122
123
124 • Ordering • Partitioning &

 Scalability
 • Fault tolerance Stream Processing Hard Parts • State Management • Time, Window &

 Out-of-order Data
 • Re-processing
125 • Ordering • Partitioning &

 Scalability
 • Fault tolerance Stream Processing Hard Parts • State Management • Time, Window &

 Out-of-order Data
 • Re-processing Simple is Beautiful
Ongoing Work (0.10+) • Beyond Java APIs • SQL support, Python client, etc • End-to-End Semantics (exactly-once) • Queryable States • … and more 126
Queryable States 127 State Real-time Analytics select Count(*), Sum(*) from “MyAgg” where windowId > now() - 10;
128 But how to get data in / out Kafka?
129
130
131
132
Take-aways • Stream Processing: a new programming paradigm 133
Take-aways • Stream Processing: a new programming paradigm • Kafka Streams: stream processing made easy 134
Take-aways • Stream Processing: a new programming paradigm • Kafka Streams: stream processing made easy 135 THANKS! Guozhang Wang | guozhang@confluent.io | @guozhangwang Visit Confluent at the Syncsort Booth (#1303), live demos @ 29th Download Kafka Streams: www.confluent.io/product
136 We are Hiring!

More Related Content

What's hot (20)

PDF
Kafka Streams: What it is, and how to use it?
confluent
 
PDF
Kafka 101 and Developer Best Practices
confluent
 
PDF
Fundamentals of Apache Kafka
Chhavi Parasher
 
PDF
Integrating Apache Kafka Into Your Environment
confluent
 
PPTX
Introduction to Apache Kafka
AIMDek Technologies
 
PDF
Apache Kafka Introduction
Amita Mirajkar
 
PDF
From Zero to Hero with Kafka Connect
confluent
 
PDF
Introduction to Apache Kafka
Shiao-An Yuan
 
PPTX
Kafka presentation
Mohammed Fazuluddin
 
PPTX
Introduction to Kafka Cruise Control
Jiangjie Qin
 
PPSX
Event Sourcing & CQRS, Kafka, Rabbit MQ
Araf Karsh Hamid
 
PDF
Parquet performance tuning: the missing guide
Ryan Blue
 
PPTX
Spring Boot+Kafka: the New Enterprise Platform
VMware Tanzu
 
PDF
Apache Kafka - Martin Podval
Martin Podval
 
PPTX
Microservices in the Apache Kafka Ecosystem
confluent
 
PDF
Dynamically Scaling Data Streams across Multiple Kafka Clusters with Zero Fli...
Flink Forward
 
PPTX
Apache Kafka Best Practices
DataWorks Summit/Hadoop Summit
 
PPTX
Kafka 101
Clement Demonchy
 
PDF
Apache kafka
NexThoughts Technologies
 
PDF
Producer Performance Tuning for Apache Kafka
Jiangjie Qin
 
Kafka Streams: What it is, and how to use it?
confluent
 
Kafka 101 and Developer Best Practices
confluent
 
Fundamentals of Apache Kafka
Chhavi Parasher
 
Integrating Apache Kafka Into Your Environment
confluent
 
Introduction to Apache Kafka
AIMDek Technologies
 
Apache Kafka Introduction
Amita Mirajkar
 
From Zero to Hero with Kafka Connect
confluent
 
Introduction to Apache Kafka
Shiao-An Yuan
 
Kafka presentation
Mohammed Fazuluddin
 
Introduction to Kafka Cruise Control
Jiangjie Qin
 
Event Sourcing & CQRS, Kafka, Rabbit MQ
Araf Karsh Hamid
 
Parquet performance tuning: the missing guide
Ryan Blue
 
Spring Boot+Kafka: the New Enterprise Platform
VMware Tanzu
 
Apache Kafka - Martin Podval
Martin Podval
 
Microservices in the Apache Kafka Ecosystem
confluent
 
Dynamically Scaling Data Streams across Multiple Kafka Clusters with Zero Fli...
Flink Forward
 
Apache Kafka Best Practices
DataWorks Summit/Hadoop Summit
 
Kafka 101
Clement Demonchy
 
Apache kafka
NexThoughts Technologies
 
Producer Performance Tuning for Apache Kafka
Jiangjie Qin
 

Viewers also liked (10)

PPTX
Introducing Kafka Streams, the new stream processing library of Apache Kafka,...
Michael Noll
 
PPTX
Apache Fink 1.0: A New Era for Real-World Streaming Analytics
Slim Baltagi
 
PPTX
Apache kafka
Rahul Jain
 
PDF
The Design of the Scalaz 8 Effect System
John De Goes
 
PPTX
Hadoop & HDFS for Beginners
Rahul Jain
 
PPTX
Kafka Streams for Java enthusiasts
Slim Baltagi
 
PDF
Building Streaming Data Applications Using Apache Kafka
Slim Baltagi
 
PPTX
Real time Analytics with Apache Kafka and Apache Spark
Rahul Jain
 
PPTX
Apache Kafka 0.8 basic training - Verisign
Michael Noll
 
PDF
reveal.js 3.0.0
Hakim El Hattab
 
Introducing Kafka Streams, the new stream processing library of Apache Kafka,...
Michael Noll
 
Apache Fink 1.0: A New Era for Real-World Streaming Analytics
Slim Baltagi
 
Apache kafka
Rahul Jain
 
The Design of the Scalaz 8 Effect System
John De Goes
 
Hadoop & HDFS for Beginners
Rahul Jain
 
Kafka Streams for Java enthusiasts
Slim Baltagi
 
Building Streaming Data Applications Using Apache Kafka
Slim Baltagi
 
Real time Analytics with Apache Kafka and Apache Spark
Rahul Jain
 
Apache Kafka 0.8 basic training - Verisign
Michael Noll
 
reveal.js 3.0.0
Hakim El Hattab
 
Ad

Similar to Introduction to Kafka Streams (20)

PDF
Stream Processing made simple with Kafka
DataWorks Summit/Hadoop Summit
 
PPTX
Exactly-once Stream Processing with Kafka Streams
Guozhang Wang
 
PDF
Kafka Summit SF 2017 - Exactly-once Stream Processing with Kafka Streams
confluent
 
PDF
Apache Kafka, and the Rise of Stream Processing
Guozhang Wang
 
PDF
Exactly-once Data Processing with Kafka Streams - July 27, 2017
confluent
 
PDF
Kafka Streams: the easiest way to start with stream processing
Yaroslav Tkachenko
 
PDF
Introducción a Stream Processing utilizando Kafka Streams
confluent
 
PDF
Richmond kafka streams intro
confluent
 
PDF
Data Streaming in Kafka
SilviuMarcu1
 
PDF
Kafka Connect and Streams (Concepts, Architecture, Features)
Kai Wähner
 
PPTX
Introduction to Kafka Streams Presentation
Knoldus Inc.
 
PPT
Kafka Explainaton
NguyenChiHoangMinh
 
PDF
Kafka streams - From pub/sub to a complete stream processing platform
Paolo Castagna
 
PDF
Kafka Streams - From the Ground Up to the Cloud
VMware Tanzu
 
PPTX
Apache Kafka Streams
Apache Kafka TLV
 
PPTX
Kick Your Database to the Curb
Bill Bejeck
 
PDF
Rethinking Stream Processing with Apache Kafka: Applications vs. Clusters, St...
Michael Noll
 
PDF
Consistency and Completeness: Rethinking Distributed Stream Processing in Apa...
Guozhang Wang
 
PDF
Apache kafka-a distributed streaming platform
confluent
 
PDF
Apache Kafka - A Distributed Streaming Platform
Paolo Castagna
 
Stream Processing made simple with Kafka
DataWorks Summit/Hadoop Summit
 
Exactly-once Stream Processing with Kafka Streams
Guozhang Wang
 
Kafka Summit SF 2017 - Exactly-once Stream Processing with Kafka Streams
confluent
 
Apache Kafka, and the Rise of Stream Processing
Guozhang Wang
 
Exactly-once Data Processing with Kafka Streams - July 27, 2017
confluent
 
Kafka Streams: the easiest way to start with stream processing
Yaroslav Tkachenko
 
Introducción a Stream Processing utilizando Kafka Streams
confluent
 
Richmond kafka streams intro
confluent
 
Data Streaming in Kafka
SilviuMarcu1
 
Kafka Connect and Streams (Concepts, Architecture, Features)
Kai Wähner
 
Introduction to Kafka Streams Presentation
Knoldus Inc.
 
Kafka Explainaton
NguyenChiHoangMinh
 
Kafka streams - From pub/sub to a complete stream processing platform
Paolo Castagna
 
Kafka Streams - From the Ground Up to the Cloud
VMware Tanzu
 
Apache Kafka Streams
Apache Kafka TLV
 
Kick Your Database to the Curb
Bill Bejeck
 
Rethinking Stream Processing with Apache Kafka: Applications vs. Clusters, St...
Michael Noll
 
Consistency and Completeness: Rethinking Distributed Stream Processing in Apa...
Guozhang Wang
 
Apache kafka-a distributed streaming platform
confluent
 
Apache Kafka - A Distributed Streaming Platform
Paolo Castagna
 
Ad

More from Guozhang Wang (11)

PDF
Consensus in Apache Kafka: From Theory to Production.pdf
Guozhang Wang
 
PDF
Exactly-Once Made Easy: Transactional Messaging Improvement for Usability and...
Guozhang Wang
 
PDF
Introduction to the Incremental Cooperative Protocol of Kafka
Guozhang Wang
 
PDF
Performance Analysis and Optimizations for Kafka Streams Applications
Guozhang Wang
 
PDF
Apache Kafka from 0.7 to 1.0, History and Lesson Learned
Guozhang Wang
 
PDF
Building Realtim Data Pipelines with Kafka Connect and Spark Streaming
Guozhang Wang
 
PDF
Building Stream Infrastructure across Multiple Data Centers with Apache Kafka
Guozhang Wang
 
PPTX
Building a Replicated Logging System with Apache Kafka
Guozhang Wang
 
PPTX
Apache Kafka at LinkedIn
Guozhang Wang
 
PPTX
Behavioral Simulations in MapReduce
Guozhang Wang
 
PPTX
Automatic Scaling Iterative Computations
Guozhang Wang
 
Consensus in Apache Kafka: From Theory to Production.pdf
Guozhang Wang
 
Exactly-Once Made Easy: Transactional Messaging Improvement for Usability and...
Guozhang Wang
 
Introduction to the Incremental Cooperative Protocol of Kafka
Guozhang Wang
 
Performance Analysis and Optimizations for Kafka Streams Applications
Guozhang Wang
 
Apache Kafka from 0.7 to 1.0, History and Lesson Learned
Guozhang Wang
 
Building Realtim Data Pipelines with Kafka Connect and Spark Streaming
Guozhang Wang
 
Building Stream Infrastructure across Multiple Data Centers with Apache Kafka
Guozhang Wang
 
Building a Replicated Logging System with Apache Kafka
Guozhang Wang
 
Apache Kafka at LinkedIn
Guozhang Wang
 
Behavioral Simulations in MapReduce
Guozhang Wang
 
Automatic Scaling Iterative Computations
Guozhang Wang
 

Recently uploaded (20)

PPTX
Knowledge Representation : Semantic Networks
Amity University, Patna
 
PPTX
Alan Turing - life and importance for all of us now
Pedro Concejero
 
PDF
methodology-driven-mbse-murphy-july-hsv-huntsville6680038572db67488e78ff00003...
henriqueltorres1
 
PPTX
Numerical-Solutions-of-Ordinary-Differential-Equations.pptx
SAMUKTHAARM
 
PPTX
Introduction to Internal Combustion Engines - Types, Working and Camparison.pptx
UtkarshPatil98
 
PDF
Basic_Concepts_in_Clinical_Biochemistry_2018كيمياء_عملي.pdf
AdelLoin
 
PPT
New_school_Engineering_presentation_011707.ppt
VinayKumar304579
 
PPTX
MODULE 05 - CLOUD COMPUTING AND SECURITY.pptx
Alvas Institute of Engineering and technology, Moodabidri
 
PDF
20ES1152 Programming for Problem Solving Lab Manual VRSEC.pdf
Ashutosh Satapathy
 
PDF
REINFORCEMENT LEARNING IN DECISION MAKING SEMINAR REPORT
anushaashraf20
 
PDF
3rd International Conference on Machine Learning and IoT (MLIoT 2025)
ClaraZara1
 
PPTX
Biosensors, BioDevices, Biomediccal.pptx
AsimovRiyaz
 
PPTX
2025 CGI Congres - Surviving agile v05.pptx
Derk-Jan de Grood
 
PDF
SERVERLESS PERSONAL TO-DO LIST APPLICATION
anushaashraf20
 
PPTX
fatigue in aircraft structures-221113192308-0ad6dc8c.pptx
aviatecofficial
 
PPT
Footbinding.pptmnmkjkjkknmnnjkkkkkkkkkkkkkk
mamadoundiaye42742
 
PPTX
darshai cross section and river section analysis
muk7971
 
PPTX
MODULE 04 - CLOUD COMPUTING AND SECURITY.pptx
Alvas Institute of Engineering and technology, Moodabidri
 
PPTX
Final Major project a b c d e f g h i j k l m
bharathpsnab
 
PPTX
MODULE 03 - CLOUD COMPUTING AND SECURITY.pptx
Alvas Institute of Engineering and technology, Moodabidri
 
Knowledge Representation : Semantic Networks
Amity University, Patna
 
Alan Turing - life and importance for all of us now
Pedro Concejero
 
methodology-driven-mbse-murphy-july-hsv-huntsville6680038572db67488e78ff00003...
henriqueltorres1
 
Numerical-Solutions-of-Ordinary-Differential-Equations.pptx
SAMUKTHAARM
 
Introduction to Internal Combustion Engines - Types, Working and Camparison.pptx
UtkarshPatil98
 
Basic_Concepts_in_Clinical_Biochemistry_2018كيمياء_عملي.pdf
AdelLoin
 
New_school_Engineering_presentation_011707.ppt
VinayKumar304579
 
MODULE 05 - CLOUD COMPUTING AND SECURITY.pptx
Alvas Institute of Engineering and technology, Moodabidri
 
20ES1152 Programming for Problem Solving Lab Manual VRSEC.pdf
Ashutosh Satapathy
 
REINFORCEMENT LEARNING IN DECISION MAKING SEMINAR REPORT
anushaashraf20
 
3rd International Conference on Machine Learning and IoT (MLIoT 2025)
ClaraZara1
 
Biosensors, BioDevices, Biomediccal.pptx
AsimovRiyaz
 
2025 CGI Congres - Surviving agile v05.pptx
Derk-Jan de Grood
 
SERVERLESS PERSONAL TO-DO LIST APPLICATION
anushaashraf20
 
fatigue in aircraft structures-221113192308-0ad6dc8c.pptx
aviatecofficial
 
Footbinding.pptmnmkjkjkknmnnjkkkkkkkkkkkkkk
mamadoundiaye42742
 
darshai cross section and river section analysis
muk7971
 
MODULE 04 - CLOUD COMPUTING AND SECURITY.pptx
Alvas Institute of Engineering and technology, Moodabidri
 
Final Major project a b c d e f g h i j k l m
bharathpsnab
 
MODULE 03 - CLOUD COMPUTING AND SECURITY.pptx
Alvas Institute of Engineering and technology, Moodabidri
 

Introduction to Kafka Streams

Editor's Notes

  • #2: Thank you.
  • #4: Well, stream processing has become widely popular today. Unlike Hadoop, Spark-like processing, which takes the bounded set of data, and only start processing until the data is completed, from a ETL process, and it can happen at a much later time than the data was originally generated, Stream processing is a real-time, continuous process for unbounded data series where the processing is usually takes a small set of record, or even one record at a time. And today, a common place to store these data streams is Kafka.
  • #5: Stream processing is a fundamental complement to capturing streams of data.
  • #10: This kind of run-as-a-service operational pattern comes from the Hadoop community.
  • #11: We think there should be an even better solution.
  • #12: No extra dependency, no enforced operational cost. In addition, it should support
  • #58: Again, in implementation such changelog streams should be compactable.
  • #59: Take all the organization's data and put it into a central place for real-time subscription. Data integration, replication, real-time stream processing.
  • #62: WAL
  • #65: Streaming on Message Pipes
  • #67: Batching: wait for all the data to be available. Reasoning about time are essential for dealing with unbounded, unordered data of varying event-time skew. Not all use cases care about event times (and if yours doesn’t, hooray! — your life is easier), but many do: billing, monitoring, anomaly detection.
  • #78: Talk about stream synchronization