Scaling big with Apache Kafka
2 likes275 views
The document discusses the architecture and functionality of Apache Kafka, emphasizing its use for handling high volumes of log data with low latency while ensuring durability and fault tolerance through features like partitioning, replication, and consumer groups. It details the concepts of message delivery semantics, including at-most-once and at-least-once delivery strategies, and introduces tools like Zookeeper and Confluent Schema Registry for schema management. Additionally, it highlights best practices for ensuring exactly-once delivery, processing failures, and monitoring Kafka systems.
Scaling big with Apache Kafka
- 2. Going big with Apache Kafka Nikolay Stoitsev - Sr. Software Engineer @ Uber
- 5. Topic Message Message Message Message Message Message
- 6. Partition - ordered, immutable sequence Message Message Message Message Message Message Message Message Message Message Message Message Message Message Message Partition 0 Partition 1 Partition 2
- 7. Offset Message Message Message Message MessagePartition 0 Offset: 1
- 8. Offset Message Message Message Message MessagePartition 0 Offset: 2
- 9. Multi Subscriber Message Message Message Message Message Offset: 5Offset: 3 Consumer: 2 Consumer: 1
- 13. Broker Broker Broker P0 P0 P0 P1 P1 P1 P2 P2 P2 Producer Producer Consumer Consumer Consumer Group 1 Consumer Consumer Consumer Consumer Group 2 Consumers
- 14. Broker Broker P0 P0 P1 P1 P2 P2 Producer Consumer Consumer Consumer Group 1 ZooKeeper Get Broker ID Update Offset
- 16. System for “collecting and delivering high volumes of log data with low latency”
- 18. Publish - Subscribe Or Kafka for interservice communication
- 19. Good throughput
- 20. Built-in partitioning, replication, and fault-tolerance
- 21. Durability
- 23. One leader for every partition Follower Leader Follower Follower Producer Consumer
- 27. Tune for lower latency ● Acknowledgement after persisted on the leader ● Can lost message on leadership changes ● At-most-once semantic
- 28. Tune for durability ● Acknowledgement after persisted on all ISR (after committed) ● No data loss ● At-least-once sematic
- 29. At-most-once cluster for logging
- 30. At-least-one cluster for message bus
- 31. Kafka as a message bus KafkaUpstream Downstream
- 33. Message queueing
- 35. Avro
- 36. KafkaUpstream Downstrea m Schema Registry Publish Schema Fetch Schema https://docs.confluent.io/current/schema-registry/docs/index.html
- 37. How to make sure something is durably stored and will be processed exactly once?
- 44. Idempotency
- 45. Idempotency + at least once delivery Process 1 Process 2Kafka Consumer
- 46. Out of the box exactly once delivery after 0.11
- 50. Order Service Kafka Payment Consumer Payment Provider Dead Letter Queue Payment Retry 0 Payment Retry 1 Dead Letter Queue
- 51. Order Service Kafka Payment Consumer Payment Provider Dead Letter Queue Payment Retry 0 Payment Retry 1 Dead Letter Queue Payment Consumer Payment Consumer
- 52. Multi Data Center Application
- 53. Regional Cluster Local Producer Regional Kafka DC1 Local Producer Regional Kafka DC2
- 54. Aggregated Cluster Local Producer Regional Kafka DC1 Local Producer Regional Kafka DC2 Kafka Replicator Aggregated Kafka DC3
- 58. How to monitor Kafka?
- 60. Detect data loss, lag and duplication
- 61. Audit Library Regional Kafka Service Kafka REST Proxy Audit Library Audit Library
- 62. Chaperone Service Regional Kafka Service Kafka REST Proxy Audit Library Audit Library Chaperone Service
- 63. Chaperone Collector Regional Kafka Service Kafka REST Proxy Audit Library Audit Library Aggregate Kafka Chaperone Service Chaperone Service Chaperone Collector DB
- 68. Summary Tune for durability Define Avro Schemas
- 69. Summary Tune for durability Define Avro Schemas Use Kafka REST Proxy
- 70. Summary Tune for durability Define Avro Schemas Use Kafka REST Proxy Add idempotency checks
- 71. Summary Tune for durability Define Avro Schemas Use Kafka REST Proxy Add idempotency checks Use Dead Letter Queue
- 72. Summary Tune for durability Define Avro Schemas Use Kafka REST Proxy Add idempotency checks Use Dead Letter Queue Monitor everything
- 73. THANK YOU! Nikolay Stoitsev Sr. Software Engineer @ Uber