Building realtime data pipeline with Apache Kafka
- 1. Building Realtime data pipeline with Apache Kafka Nagarajan Developer, ThoughtWorks
- 2. A pipeline?
- 3. Data pipeline ● Data flow between different systems/components ● Loosely coupled ● Automated ● Scalable ● Error recovery
- 4. Usecase Image you’ve a web application, and you want to see number of visitors per page on different time durations
- 9. Characteristics required ● High throughput ingestion ● Fault tolerant storage ● Highly available ● Scalable ● Support for concurrent processing and ordering guarantee
- 10. Enter Kafka Apache Kafka® is a distributed streaming platform that: ● Publishes and subscribes to streams of records, similar to a message queue or enterprise messaging system. ● Stores streams of records in a fault-tolerant durable way. ● Helps to process streams of records as they occur. Kafka is a distributed, partitioned, replicated commit log service. It provides the functionality of a messaging system with unique design.
- 11. Kafka APIs
- 12. Messaging system ● Queue ● publish-subscribe
- 14. Partition and message ordering Each partition is an ordered, immutable sequence of records that is continually appended to a structured commit log Records in the partitions are each assigned a sequential id number called the offset that uniquely identifies each record within the partition How does partition assigned?
- 15. Data retention ● Log retention by time ○ log.retention.ms (minutes/hours) ● Log retention by size ○ log.retention.bytes ● Log segments ○ log.segment.bytes ○ log.segment.ms ● Log compaction
- 16. Log compaction
- 17. Broker, Cluster and Zookeeper
- 18. Replication ● Replica types ○ Leader replica ○ Follower replica ● In Sync Replicas(ISR) ○ min.insync.replicas ● Replication factor
- 19. Producer ● acks ● Batching ○ batch.size ○ linger.ms ● Send Async ● Load balancing
- 20. Producer flow
- 21. Consumer and Consumer groups
- 22. Consumer and Consumer groups ● Push vs Pull ● Consumer position - offset maintenance ○ Auto commit ○ __consumer_offsets ● Replay ● Partition assignment ○ Range ○ Round robin
- 23. Delivery semantics ● At most once ● At least once ● Exactly once ○ Idempotency (exactly once produce) ○ Transactions (end to end exactly once) - only with Kafka streams
- 24. Schemas ● Decouple Producers and Consumers ● Avro format ● Schema Registry ● Evolution and compatibility
- 25. Fire fighting stories ● Consumer offsets retention ● Increase number of partitions ● Record without key
- 28. Advanced ● Security ● Configuration ● Monitoring ● Transactions ● Kafka connect ● Schema registry ● Kafka streams ● KSQL