Consistency and Completeness: Rethinking Distributed Stream Processing in Apache Kafka

Guozhang Wang Lei Chen Ayusman Dikshit Jason Gustafson
Boyang Chen Matthias J. Sax John Roesler Sophie Blee-Goldman Bruno Cadonna
Apurva Mehta Varun Madan Jun Rao
Consistency and Completeness
Rethinking Distributed Stream Processing in Apache Kafka

Outline
• Stream processing (with Kafka): correctness challenges
• Exactly-once consistency with failures
• Completeness with out-of-order data
• Use case and conclusion
2

3
Stream Processing
• A different programming paradigm
• .. that brings computation to unbounded data
• .. but not necessarily transient, approximate, or lossy

4
• Persistent Buffering
• Logical Ordering
• Scalable “source-of-truth”
Kafka: Streaming Platform

Kafka Concepts: the Log
4 5 5 7 8 9 10 11 12
...
Producer Write
Consumer1 Reads
(offset 7)
Consumer2 Reads
(offset 10)
Messages
3
5

Topic 1
Topic 2
Partitions
Producers
Producers
Consumers
Consumers
Brokers
Kafka Concepts: the Log
6

Topic 1
Topic 2
Partitions
Producers
Producers
Consumers
Consumers
Brokers
7
High-Availability: Must-have

Topic 1
Topic 2
Partitions
Producers
Producers
Consumers
Consumers
Brokers
7
High-Availability: Must-have
[VLDB 2015]

8
Process
State
Kafka Topic A
Kafka Topic B
Kafka Topic C
Kafka Topic D
Your App
Stream Processing with Kafka

9
Process
State
Kafka Topic A
Kafka Topic B
Kafka Topic C
Kafka Topic D
Your App

9
Process
State
Kafka Topic A
Kafka Topic B
Kafka Topic C
Kafka Topic D
ack
ack
Your App

9
Process
State
Kafka Topic A
Kafka Topic B
Kafka Topic C
Kafka Topic D
ack
ack
commit
Your App

10
Kafka Topic A
Kafka Topic B
Kafka Topic C
Kafka Topic D
State
Process
Streams App
Duplicate Results on Failures

10
Kafka Topic A
Kafka Topic B
Kafka Topic C
Kafka Topic D
ack
ack
State
Process
Streams App

10
Kafka Topic A
Kafka Topic B
Kafka Topic C
Kafka Topic D
State
Streams App

11
State
Process
Kafka Topic A
Kafka Topic B
Kafka Topic C
Kafka Topic D
Streams App

11
State
Process
Kafka Topic A
Kafka Topic B
Kafka Topic C
Kafka Topic D
State
Streams App

12

12
2
2
3
3
4
4

12
2
2
3
3
4
4
Duplicates propagated downstream and emitted externally

13
Challenge #1: Consistency (a.k.a. exactly-once)
An correctness guarantee for stream processing,
.. that for each received record,
.. its process results will be reflected exactly once,
.. even under failures

It’s all about Time
• Event-time (when a record is created)
• Processing-time (when a record is processed)
14

Event-time 4 5 6 1 2 3 7 8
Processing-time 1977 1980 1983 1999 2002 2005 2015 2017
15
P
H
A
N
T
O
M
M
E
N
A
C
E
A
T
T
A
C
K
O
F
T
H
E
C
L
O
N
E
S
R
E
V
E
N
G
E
O
F
T
H
E
S
I
T
H
A
N
E
W
H
O
P
E
T
H
E
E
M
P
I
R
E
S
T
R
I
K
E
S
B
A
C
K
R
E
T
U
R
N
O
F
T
H
E
J
E
D
I
T
H
E
F
O
R
C
E
A
W
A
K
E
N
S
Out-of-Order
T
H
E
L
A
S
T
J
E
D
I

Event-time 4 5 6 1 2 3 7 8
Processing-time 1977 1980 1983 1999 2002 2005 2015 2017
15
P
H
A
N
T
O
M
M
E
N
A
C
E
A
T
T
A
C
K
O
F
T
H
E
C
L
O
N
E
S
R
E
V
E
N
G
E
O
F
T
H
E
S
I
T
H
A
N
E
W
H
O
P
E
T
H
E
E
M
P
I
R
E
S
T
R
I
K
E
S
B
A
C
K
R
E
T
U
R
N
O
F
T
H
E
J
E
D
I
T
H
E
F
O
R
C
E
A
W
A
K
E
N
S
Out-of-Order
T
H
E
L
A
S
T
J
E
D
I
Incomplete results produced due to time skewness

16
Challenge #2: Completeness (with out-of-order data)
An correctness guarantee for stream processing,
.. that even with out-of-order data streams,
.. incomplete results would not be delivered

17
Blocking + Checkpointing
One stone to kill all birds?
• Block processing and result emitting until complete
• Hard trade-offs between latency and correctness
• Depend on global blocking markers

18
A Log-based Approach:
• Leverage on persistent, immutable, ordered logs
• Decouple consistency and completeness handling

Kafka Streams
• New client library beyond producer and consumer
• Powerful yet easy-to-use
• Event time, stateful processing
• Out-of-order handling
• Highly scalable, distributed, fault tolerant
• and more..
19

20
Anywhere, anytime
Ok. Ok. Ok. Ok.

Streams DSL and KSQL
21
CREATE STREAM fraudulent_payments AS
SELECT * FROM payments
WHERE fraudProbability > 0.8;
val fraudulentPayments: KStream[String, Payment] = builder
.stream[String, Payment](“payments-kafka-topic”)
.filter((_ ,payment) => payment.fraudProbability > 0.8)
fraudulentPayments.to(“fraudulent-payments-topic”)
[EDBT 2019]

22
Kafka Topic B Kafka Topic A
P1
P2
P1
P2
Processing in Kafka Streams

23
Kafka Topic B Kafka Topic A
Processor Topology
P1
P2
P1
P2

24
Kafka Topic A
Kafka Topic B
P1
P2
P1
P2

MyApp.2
MyApp.1
Kafka Topic B
Task2
Task1
25
Kafka Topic A
State State
P1
P2
P1
P2

MyApp.2
MyApp.1
Kafka Topic B
Task2
Task1
25
Kafka Topic A
State State

MyApp.2
MyApp.1
Kafka Topic B
Task2
Task1
26
Kafka Topic A
State State

MyApp.2
MyApp.1
Kafka Topic B
Task2
Task1
26
Kafka Topic A
State State
Kafka Changelog Topic

27
Process
State
Kafka Topic C
Kafka Topic D
ack
ack
Kafka Topic A
Kafka Topic B
commit
Exactly-Once with Kafka

27
Process
State
Kafka Topic C
Kafka Topic D
ack
ack
Kafka Topic A
Kafka Topic B
commit
• Offset commit for source topics

27
Process
State
Kafka Topic C
Kafka Topic D
ack
ack
Kafka Topic A
Kafka Topic B
commit
• State update on processor

27
Process
State
Kafka Topic C
Kafka Topic D
ack
ack
Kafka Topic A
Kafka Topic B
commit
• Acked produce to sink topics

27

28

29
All or Nothing

30
Exactly-Once with Kafka Streams
All or Nothing

30

31
• A batch of records sent to the offset topic

32
• A batch of records sent to changelog topics

33
• A batch of records sent to sink topics

34
All or Nothing
• A batch of records sent to sink topics

35
Input Topic
State
Process
Streams
Changelog Topic
Output Topic
Offset Topic
producer.initTxn();
<-
Txn Coordinator
Txn Log Topic

35
Input Topic
State
Process
Streams
Changelog Topic
Output Topic
Offset Topic
producer.initTxn();
<-
Txn Coordinator
Txn Log Topic
register txn.id

35
Input Topic
State
Process
Streams
Changelog Topic
Output Topic
Offset Topic
producer.initTxn();
<-
Txn Coordinator
Txn Log Topic
txn.id -> empty ()
register txn.id

36
Input Topic
Process
Streams
Changelog Topic
Output Topic
Offset Topic
producer.initTxn();
try {
producer.beginTxn();
recs = consumer.poll();
<-
}
Txn Coordinator
Txn Log Topic
State
txn.id -> empty ()

37
Input Topic
State
Process
Streams
Changelog Topic
Output Topic
Offset Topic
producer.initTxn();
try {
for (Record rec <- recs) {
// process ..
}
}
Txn Coordinator
Txn Log Topic
txn.id -> empty ()

38
Input Topic
State
Process
Streams
Changelog Topic
Output Topic
Offset Topic
producer.initTxn();
try {
// process ..
producer.send(“output”, ..);
producer.send(“changelog”, ..);
}
}
Txn Coordinator
Txn Log Topic
txn.id -> empty ()

38
Input Topic
State
Process
Streams
Changelog Topic
Output Topic
Offset Topic
producer.initTxn();
try {
// process ..
}
}
Txn Coordinator
Txn Log Topic
add partition
txn.id -> empty ()

38
Input Topic
State
Process
Streams
Changelog Topic
Output Topic
Offset Topic
producer.initTxn();
try {
// process ..
}
}
Txn Coordinator
Txn Log Topic
txn.id -> ongoing
(output, changelog)
add partition

39
Input Topic
State
Process
Streams
Changelog Topic
Output Topic
Offset Topic
producer.initTxn();
try {
// process ..
producer.sendOffsets(“input”, ..);
}
}
Txn Coordinator
Txn Log Topic
txn.id -> ongoing
(output, changelog)

39
Input Topic
State
Process
Streams
Changelog Topic
Output Topic
Offset Topic
producer.initTxn();
try {
// process ..
}
}
Txn Coordinator
Txn Log Topic
add offsets
txn.id -> ongoing
(output, changelog)

39
Input Topic
State
Process
Streams
Changelog Topic
Output Topic
Offset Topic
producer.initTxn();
try {
// process ..
}
}
Txn Coordinator
Txn Log Topic
add offsets
txn.id -> ongoing
(output, changelog, offset)

40
Input Topic
State
Process
Streams
Changelog Topic
Output Topic
Offset Topic
producer.initTxn();
try {
// process ..
}
producer.commitTxn();
}
Txn Coordinator
Txn Log Topic
txn.id -> ongoing

40
Input Topic
State
Process
Streams
Changelog Topic
Output Topic
Offset Topic
producer.initTxn();
try {
// process ..
}
}
Txn Coordinator
Txn Log Topic
prepare commit
txn.id -> ongoing

40
Input Topic
State
Process
Streams
Changelog Topic
Output Topic
Offset Topic
producer.initTxn();
try {
// process ..
}
}
Txn Coordinator
Txn Log Topic
prepare commit
txn.id -> prepare-commit

41
Input Topic
State
Process
Streams
Changelog Topic
Output Topic
Offset Topic
producer.initTxn();
try {
// process ..
}
}
Txn Coordinator
Txn Log Topic
txn.id -> prepare-commit

41
Input Topic
State
Process
Streams
Changelog Topic
Output Topic
Offset Topic
producer.initTxn();
try {
// process ..
}
}
Txn Coordinator
Txn Log Topic
txn.id -> complete-commit

42
Input Topic
State
Process
Streams
Changelog Topic
Output Topic
Offset Topic
producer.initTxn();
try {
// process ..
}
} catch (KafkaException e) {
producer.abortTxn();
}
Txn Coordinator
Txn Log Topic
txn.id -> ongoing

42
Input Topic
State
Process
Streams
Changelog Topic
Output Topic
Offset Topic
producer.initTxn();
try {
// process ..
}
}
Txn Coordinator
Txn Log Topic
prepare abort
txn.id -> ongoing

42
Input Topic
State
Process
Streams
Changelog Topic
Output Topic
Offset Topic
producer.initTxn();
try {
// process ..
}
}
Txn Coordinator
Txn Log Topic
txn.id -> prepare-abort
prepare abort

43
Input Topic
State
Process
Streams
Changelog Topic
Output Topic
Offset Topic
producer.initTxn();
try {
// process ..
}
}
Txn Coordinator
Txn Log Topic
txn.id -> prepare-abort

43
Input Topic
State
Process
Streams
Changelog Topic
Output Topic
Offset Topic
producer.initTxn();
try {
// process ..
}
}
Txn Coordinator
Txn Log Topic
txn.id -> complete-abort

44
Input Topic
State
Process
Streams
Changelog Topic
Output Topic
Offset Topic
producer.initTxn();
try {
// process ..
}
}
Txn Coordinator
Txn Log Topic
txn.id -> complete-abort

45
State
Process
State
Process
State
Process
Exactly-Once with Failures
Kafka
Kafka Streams
Kafka Changelog
Kafka

46
State
Process
State
Process
State
Process
Kafka
Kafka Streams
Kafka Changelog
Kafka

46
State
Process
State
Process
Kafka
Kafka Streams
Kafka Changelog
Kafka

47
State
Process
State
Process
Process
Kafka
Kafka Streams
Kafka Changelog
Kafka
State

48
State
Process
State
Process
State
Process
Kafka
Kafka Streams
Kafka Changelog
Kafka

49
State
Process
State
Process
State
Process
Kafka
Kafka Streams
Kafka Changelog
Kafka

50
What about Completeness?
• Option 1: defer emitting output and committing txn
• Effectively coupling completeness with consistency
• Increased end-to-end processing latency
• Option 2: emitting output early when possible
• Not try to prevent incompleteness via coordination
• Instead, compensate when out-of-order data happens

51
2
2
3
3
4
4
Remember the Logs
• upstream-downstream communication can be replayed
• Emitted records are naturally ordered by offsets

52
Ordering and Monotonicity
• Stateless (filter, mapValues)
• Order-agnostic: no need to block on emitting
• Stateful (join, aggregate)
• Order-sensitive: current results depend on history
• Whether block emitting results depend on output types

KStream = interprets data as record stream
~ think: “append-only”
KTable = data as changelog stream
~ continuously updated materialized view
53

54
alice eggs bob bread alice milk
alice lnkd bob googl alice msft
User purchase history
User employment profile
KStream
KTable

55
time
“Alice bought eggs.”
“Alice is now at LinkedIn.”
KStream
KTable

56
time
“Alice bought eggs and milk.”
“Alice is now at LinkedIn
Microsoft.”
KStream
KTable

57
time
(Alice: A/null)
alice 10
alice A alice B
“do not emit”
KStream.leftJoin(KStream)
-> KStream
KTable.leftJoin(KTable)
-> KTable

58
time
alice 10
alice A alice B
(Alice: A/null)
(Alice: A/10)
“do not emit”
(Alice: A/10)
-> KStream
-> KTable

59
time
alice 10
alice A alice B
(Alice: A/null)
(Alice: A/10)
(Alice: B/10)
“do not emit”
(Alice: A/10)
(Alice: B/10)
-> KStream
-> KTable

60
time
(Alice: null)
(Alice: null)
alice A/null A/10 alice B/10
alice
-> KStream
.aggregate()
-> KTable
-> KTable
.aggregate()
-> KTable

61
time
(Alice: null 10)
(Alice: null+10)
alice
-> KStream
.aggregate()
-> KTable
-> KTable
.aggregate()
-> KTable

62
time
(Alice: 10 10)
(Alice: null+10+10)
-> KStream
.aggregate()
-> KTable
-> KTable
.aggregate()
-> KTable
alice

62
time
(Alice: 10 10)
(Alice: null+10+10)
-> KStream
.aggregate()
-> KTable
-> KTable
.aggregate()
-> KTable
alice
[BIRTE 2015]

63
Use Case: Bloomberg Real-time Pricing

64

64
• One billion plus market events / day
• 160 cores / 2TB RAM deployed on k8s
• Exactly-once for market data stateful stream processing

• Apache Kafka: persistent logs to achieve correctness
65
Take-aways

• Transactional log appends for exactly-once
66
Take-aways

• Non-blocking output with revisions to handle out-of-order data
67
Take-aways

68
Take-aways
Guozhang Wang | guozhang@confluent.io | @guozhangwang
Read the full paper at: https://cnfl.io/sigmod

68
Take-aways
THANKS!
Guozhang Wang | guozhang@confluent.io | @guozhangwang
Read the full paper at: https://cnfl.io/sigmod

Ongoing Work (3.0+)
• Scalability improvements
• Consistent state query serving
• Further reduce end-to-end latency
• … and more
70

Consistency and Completeness: Rethinking Distributed Stream Processing in Apache Kafka

More Related Content

What's hot (20)

Similar to Consistency and Completeness: Rethinking Distributed Stream Processing in Apache Kafka (20)

More from Guozhang Wang (11)

Recently uploaded (20)

Consistency and Completeness: Rethinking Distributed Stream Processing in Apache Kafka