Autonomous workload rebalancing in kafka
- 1. Autonomous workload rebalancing in Kafka Indrajeet Kumar Site Reliability Engineer - LinkedIn
- 2. Agenda ● Workload distribution problem ● Manual - Built-in tools ● Semi-automated - Kafka-assigner ● Autonomous - Cruise Control
- 3. Workload distribution problem ● Important for Distributed Systems ● Harder to work around with Stateful systems
- 4. Kafka Overview App 2 App 1 App 3 Topic A Topic C Topic D Topic B Kafka
- 5. Kafka Overview P 1 P 2 P 3 App 1 Topic A P1 Broker X P2 Broker Y P3 Broker Z R P3 R P2 R P1
- 6. Workload in Kafka ○ Leader Partitions ○ Total Partitions ○ Partition Sizes P1 Broker X P2 Broker Y P3 Broker Z R P3 R P2 R P1
- 7. Workload in Kafka ○ Leader Partitions ○ Total Partitions ○ Partition Sizes P1 Broker X P2 Broker Y P3 Broker Z R P3 R P2 R P1
- 8. Workload in Kafka ○ Leader Partitions ○ Total Partitions ○ Partition Sizes P1 Broker X P2 Broker Y P3 Broker Z R P3 R P2 R P1
- 9. Workload in Kafka ○ Leader Partitions ○ Total Partitions ○ Partition Sizes P1 Broker X P2 Broker Y P3 Broker Z R P3 R P2 R P1
- 10. Workload distribution problem - Some causes ● Major factors which affect workload balance are: ○ Bad partition distribution ○ Hard host failures ○ Soft host failures ○ Traffic patterns
- 11. ● Rebalance the partitions! ○ Disk usage ○ Network usage ○ Number of partitions ○ Partition leadership count Kafka workload distribution - Solution
- 12. Usual operations in Kafka ● Preferred Leader Election ● Partition rebalance ● Bump Partition counts ● Add/Remove brokers
- 14. Kafka at LinkedIn ● 4.5 Trillion messages a day ● 2500+ kafka brokers ● 1 PB In ● 3.9 PB Out
- 15. Kafka admin utilities ● Out of the box tools: ○ bin/kafka-reassign-partitions.sh ○ bin/kafka-preferred-replica-election.sh
- 16. Example run of built-in tools ● Rebalancing Partitions: . ● <Example of kafka-preferred-replica-election.sh>
- 17. ● Manual ● Less optimal ● Slow Problems with stock tools
- 18. Kafka Assigner
- 19. Kafka assigner ● High level administrative commands ● Under the hood, it uses the ‘kafka-utils/bin/’ scripts ● It also allows to do complex rebalances with multiple goals
- 20. Kafka assigner
- 22. Case of URPs
- 23. Kafka assigner ● Pros: ○ High level admin commands ○ Simple to use ○ Allows chaining rebalance goals ○ Easy to remove all partitions from a broker
- 24. Kafka assigner ● Cons: ○ Where did you run it? ○ In-optimal balances in certain cases ○ Needs manual invocation and supervision
- 25. Cruise Control
- 26. Cruise Control ● Central System ● Complete live health of the cluster ● Manual/Automatic management of workload
- 27. Design User Kafka Cruise Control R E S T A P I Analyzer Failure Detector Workload Monitor KAFKA Goal Executor
- 30. Cruise Control
- 31. CC setup requirements ● Kafka > 0.11.0.0 ● Drop in jar
- 32. ● Resource utilization tracking ● Multi-goal rebalance ● Anomaly detection ● Admin operations Features already built-in
- 33. How is CC doing? ● Save SRE’s time to debug/fix kafka workload issues ● Very fast operations ● Central place to look at for globally distributed teams ● Self-heal !!
- 34. Resources Kafka shipped admin-tools: https://github.com/apache/kafka/tree/trunk/bin Kafka Assigner: https://github.com/linkedin/kafka-tools/wiki/Kafka-Assigner Cruise Control: https://github.com/linkedin/cruise-control Connect with me: https://www.linkedin.com/in/indrajeetkm/
- 35. Questions