How to tune Kafka® for production
9 likes1,541 views
This document discusses various configurations and techniques for optimizing Kafka for latency, throughput, durability, and availability. For latency, it recommends small batches, no compression, low replication guarantees, and fetching data as soon as possible. For throughput, it suggests batching, compression, increasing memory, and parallelizing with consumer groups. For durability, it highlights replication, idempotent producers, and exactly-once processing. And for availability, it notes the importance of replicas and log recovery configurations. The document provides guidance on tuning Kafka deployments for different performance objectives.
![Durability: Exactly Once Semantic
producer.initTransactions();
[...]
while (true) {
ConsumerRecords records = consumer.poll(Long.MAX_VALUE);
producer.beginTransaction();
for (ConsumerRecord record : records)
producer.send(producerRecord(“outputTopic”, record));
producer.sendOffsetsToTransaction(currentOffsets(consumer), group);
producer.commitTransaction();
}
Source: Confluent
1. Idempotent
Producer
2. Multiple
Partition
Transactions
#atix #streamingworkshop](https://image.slidesharecdn.com/streamingworkshop-190724190712/85/How-to-tune-Kafka-for-production-43-320.jpg)
How to tune Kafka® for production
- 1. v = 346,39 m/s
- 2. v = 346,39 m/s Speed of sound at 25 °C v = 349,29 m/s Speed of sound at 30 °C front to audience: 2 m latency: 5.7 ms latency (back of room): 15 ms #atix #streamingworkshop
- 3. v = 346,39 m/s Speed of sound at 25 °C v = 349,29 m/s Speed of sound at 30 °C front to audience: 2 m latency: 5.7 ms latency (back of room): 15 ms #atix #streamingworkshop
- 4. v = 346,39 m/s Speed of sound at 25 °C v = 349,29 m/s Speed of sound at 30 °C front to audience: 2 m latency: 5.7 ms latency (back of room): 15 ms #atix #streamingworkshop
- 5. v = 346,39 m/s Speed of sound at 25 °C v = 349,29 m/s Speed of sound at 30 °C front to audience: 2 m latency: 5.7 ms latency (back of room): 15 ms #atix #streamingworkshop
- 6. v = 346,39 m/s Speed of sound at 25 °C v = 349,29 m/s Speed of sound at 30 °C front to audience: 2 m latency: 5.7 ms latency (back of room): 15 ms #atix #streamingworkshop
- 7. take home message: I am faster than my ISP! Source: speedcheck.org #atix #streamingworkshop
- 8. How to tune Kafka for production Dr. Bernhard Hopfenmüller 24. Juli 2019 #atix #streamingworkshop
- 9. What do you need? Latency Throughput Durability Availability ??? IoT #atix #streamingworkshop
- 10. whoami Bernhard Hopfenmüller Senior IT Consultant @ ATIX AG current latency 8 ms [email protected] github.com/Fobhep Twitter: @fobhep IRC: Fobhep #atix #streamingworkshop
- 12. Throughput: Batching increase batch size increase linger time #atix #streamingworkshop
- 13. Throughput: Compression Compress FULL batches lz4, snappy, zstd, gzip be consistent - avoid recompression!! #atix #streamingworkshop
- 14. Throughput: Replication Guarantees did my message arrive? acks=1 (default) #atix #streamingworkshop
- 15. Throughput: Memory Increase producer buffer memory Increase consumer fetch size #atix #streamingworkshop
- 16. Throughput: Consumer Groups Parallelize data procession #atix #streamingworkshop
- 17. Throughput: JVM -Xms6g -Xmx6g -XX:MetaspaceSize=96m -XX:+UseG1GC -XX:MaxGCPauseMillis=20 -XX:InitiatingHeapOccupancyPercent=35 -XX:G1HeapRegionSize=16M -XX:MinMetaspaceFreeRatio=50 -XX:MaxMetaspaceFreeRatio=80 Too long JVM GC pause time could cause Zookeeper session timeout impact throughput #atix #streamingworkshop
- 18. Throughput: Partitions More Partition == Higher Troughput?! yes, but! More Open File Handlers #atix #streamingworkshop
- 19. Throughput: Partitions More Partition == Higher Troughput?! yes, but! More Open File Handlers More Memory needed for Producer and Consumer #atix #streamingworkshop
- 20. Throughput: Summary Producer: batch.size: 100000 – 200000 (default 16384) linger.ms: 10 – 100 (default 0) compression.type: lz4 (default none, i.e., no compression) acks: 1 (default 1) buffer.memory: increase according to partitions (default 33554432) Consumer: fetch.min.bytes: ~100000 (default 1) #atix #streamingworkshop
- 22. Latency: Batching small batches! linger.ms=0 not! no batching #atix #streamingworkshop
- 23. Latency: Compression no compression reduces bandwidth utilization less cpu load but! benchmark, codecs might be good after all #atix #streamingworkshop
- 24. Latency: Replication Guarantees did my message arrive? #atix #streamingworkshop
- 25. Latency: Replication Guarantees acks=1 (0 if you are brave) #atix #streamingworkshop
- 26. Latency: Memory fetch.min.bytes=1 (fetch data asap) #atix #streamingworkshop
- 27. Latency: Kafka Streams - External Queries Source: Confluent Query external databases high latency #atix #streamingworkshop
- 28. Latency: Kafka Streams - Table Stream Duality Source: Confluent A table is a stream is a table ... #atix #streamingworkshop
- 29. Latency: Kafka Streams - Local Table Joins Source: Confluent Include external data via Connect join Tables locally with streams decrease latency #atix #streamingworkshop
- 30. Latency: Kafka Streams - Topologies Source: Confluent Input from Kafka Operations (filter map aggregation join) Back to Kafka #atix #streamingworkshop
- 31. Latency: Kafka Streams - Topologies Source: Confluent Repartitioning may occur increase latency #atix #streamingworkshop
- 32. Latency: Kafka Streams - Topologies config.setProperty( StreamsConfig.TOPOLOGY_OPTIMIZATION, StreamsConfig.OPTIMIZE); Avoid unnecessary repartitioning decrease latency #atix #streamingworkshop
- 33. Latency: Partitions Source: Confluent More Partition == Higher Troughput?! yes, but! broker has single thread for replication (default) latency might increase limit topics per broker increase num.replica.fetchers #atix #streamingworkshop
- 34. Latency: Summary Producer: linger.ms: 0 (default) compression.type: none (default) acks:1 (default) Consumer: fetch.min.bytes=1 Streams: StreamsConfig.TOPOLOGY_OPTIMIZATION StreamsConfig.OPTIMIZE use kafka connect and local joins Broker: num.replica.fetchers: increase (default 1) #atix #streamingworkshop
- 36. Durability: Producer Replication Source: Confluent choose a higher replication factor don’t forget internal topics! __consumer_offsets internal streams topics exactly once semantics #atix #streamingworkshop
- 37. Durability: Producer Acks acks=all #atix #streamingworkshop
- 38. Durability: Producer Retries Source: Confluent retries duplicates ordering problems #atix #streamingworkshop
- 39. Durability: Producer Idempotency Source: Confluent produce idempotent! #atix #streamingworkshop
- 40. Durability: Delivery Semantics at-least-once delivery at-most-once delivery exactly-once delivery (Kafka only!) #atix #streamingworkshop
- 41. Durability: Exactly Once Semantic ( Send Data with PID and increasing seq Broker: Check if seq > seq_old write only if true ) 1. Idempotent Producer #atix #streamingworkshop
- 42. Durability: Exactly Once Semantic ( Send Data Read Data (Write to Commit Log) ) Either write all of that or none Source: Confluent 1. Idempotent Producer 2. Multiple Partition Transactions #atix #streamingworkshop
- 43. Durability: Exactly Once Semantic producer.initTransactions(); [...] while (true) { ConsumerRecords records = consumer.poll(Long.MAX_VALUE); producer.beginTransaction(); for (ConsumerRecord record : records) producer.send(producerRecord(“outputTopic”, record)); producer.sendOffsetsToTransaction(currentOffsets(consumer), group); producer.commitTransaction(); } Source: Confluent 1. Idempotent Producer 2. Multiple Partition Transactions #atix #streamingworkshop
- 44. Durability: Streams EOS StreamsConfig.PROCESSING_GUARANTEE_CONFIG: StreamsConfig.EXACTLY_ONCE StreamsConfig.REPLICATION_FACTOR_CONFIG: 3 (default 1) EOS for Streams very simple don’t forget replication of streams topics #atix #streamingworkshop
- 45. Durability: Consumer no auto commit! enable.auto.commit false for EOS (isolation.level = read_commited) #atix #streamingworkshop
- 46. Durability: Replica min.insync.replica > 1 #atix #streamingworkshop
- 47. Durability: Summary Producer: replication.factor=3 (topic override available) acks=all (default 1) enable.idempotence=true (default false), to handle message duplication and ordering Consumer: enable.auto.commit=false (default true) isolation.level=read_committed (for EOS transactions) #atix #streamingworkshop
- 48. Durability: Summary Streams: StreamsConfig.REPLICATION_FACTOR_CONFIG: 3 (default 1) StreamsConfig.PROCESSING_GUARANTEE_CONFIG: StreamsConfig.EXACTLY_ONCE Broker: default.replication.factor=3 (default 1) auto.create.topics.enable=false (default true) min.insync.replicas=2 (default 1); topic override available broker.rack: rack of the broker (default null) #atix #streamingworkshop
- 50. Availability: Replica min.insync.replicas = 1 unclean.leader.election.enable = true #atix #streamingworkshop
- 51. Availability: Log Recovery broker start: scan all log files to sync usually one thread per data dir increase to number of dirs in log.files #atix #streamingworkshop
- 52. Availability: Stream State Restoration Source: Confluent store data for stateful stream ops create copies of state for reusing num.standby.replicas > 0 #atix #streamingworkshop
- 53. Availability: Summary Consumer: session.timeout.ms: as low as feasible (default 10000) Streams: StreamsConfig.NUM_STANDBY_REPLICAS_CONFIG: 1 or more(default 0) Broker: unclean.leader.election.enable=true (default false); topic override available min.insync.replicas=1 (default 1); topic override available num.recovery.threads.per.data.dir: number of directories in log.dirs (default 1) #atix #streamingworkshop
- 54. And now?: Next steps 1. Decide what you want Repeat: 2. Benchmark - Control Center! 3. Act on results 4. MONITOR! - Control Center! #atix #streamingworkshop
- 55. And now?: Sources and Material What is Kafka https://www.heise.de/select/i- x/2019/4/1553935521978043 Kafka Optimization https://www.confluent.io/white- paper/optimizing-your-apache-kafka-deployment/ Kafka Partition number https://www.confluent.io/blog/how-choose- number-topics-partitions-kafka-cluster RTFM https://docs.confluent.io/current/administer.html #atix #streamingworkshop
- 56. And now?: Talk to us Datacenter Automation Deploy ConfigureRelease #atix #streamingworkshop
- 57. Let’s optimize our latency! #atix #streamingworkshop