Apache Kafka 0.11 の Exactly Once Semantics
This document discusses exactly once semantics in Apache Kafka 0.11. It provides an overview of how Kafka achieved exactly once delivery between producers and consumers. Key points include: - Kafka 0.11 introduced exactly once semantics with changes to support transactions and deduplication. - Producers can write in a transactional fashion and receive acknowledgments of committed writes from brokers. - Brokers store commit markers to track the progress of transactions and ensure no data loss during failures. - Consumers can read from brokers in a transactional mode and receive data only from committed transactions, guaranteeing no duplication of records. - This allows reliable message delivery semantics between producers and consumers with Kafka acting as
Apache Kafka 0.11 の Exactly Once Semantics
- 1. Apache Kafka 0.11 Exactly Once Semantics / @laclefyoshi / [email protected]
- 2. • • Kafka • Exactly Once : • 2
- 3. • 2011/04 • 2015/09 • • Druid (KDP, 2015) • RDB NoSQL ( , 2016; : HBase ) • ESP8266 Wi-Fi IoT (KDP, 2016) • • (WebDB Forum 2014) • Spark Streaming (Spark Meetup December; 2015) • Kafka AWS Kinesis (Apache Kafka Meetup Japan #1; 2016) • (FutureOfData; 2016) • Queryable State for Kafka Streams (Apache Kafka Meetup Japan #2; 2016) • Apache Spark ( Geek Night #11; 2016) • (BigData-JAWS #6; 2017) 3
- 4. Kafka
- 5. Kafka 5
- 8. Kafka → → 8
- 9. Kafka At Least Once At Most Once 9
- 10. Kafka At Least Once At Most Once Exactly Once 9
- 11. Exactly Once :
- 12. Kafka 0.11.0.0 • 2017/06/28 • KIP-129 → KAFKA-4923: Add Exactly-Once Semantics to Streams • https://issues.apache.org/jira/browse/KAFKA-4923 • Confluent Blog: Exactly-once Semantics are Possible: Here’s How Kafka Does it • https://www.confluent.io/blog/exactly-once-semantics-are-possible- heres-how-apache-kafka-does-it/ • “In the following posts in this series, we will go into more details …” • Exactly Once Delivery and Transactional Messaging in Kafka • https://docs.google.com/document/d/ 11Jqy_GjUGtdXJK94XGsEIK7CP1SnQGdp2eF0wSw9ra8/ • KIP-98 → KAFKA-5059: Implement Transactional Coordinator 11
- 14. Kafka 13
- 15. Kafka : Exactly Once 14
- 17. Producer Exactly Once 1. 2. 3. x N 4. → 16
- 18. Producer Exactly Once 1. 2. 3. x N 4. → → → 1 17
- 19. Producer Exactly Once 5.Broker Commit Marker COMMITTED ABORTED ↓ ↓ 18
- 21. Consumer Exactly Once 6. ↓ READ_COMMITTED ↓ READ_UNCOMMITTED 20
- 22. Consumer Exactly Once 7. 8. - COMMITTED - ABORTED Transaction List 21
- 23. • Kafka API RecordBatch • Kafka < 0.11 Message MessageSet API • Kafka 0.11 MessageSet/Message RecordBatch/Record • • Producer Broker Consumer ID PID Producer ID 22
- 24. Exactly Once • : Exactly Once • Producer • Broker • Commit Marker • Consumer • → OSS
- 25. • Exactly Once • Producer Broker • Consumer • 3% • • Confluent • OSS Kafka • Exactly Once 24