Metrics are Not Enough: Monitoring Apache Kafka / Gwen Shapira (Confluent)
Download as PPTX, PDF5 likes689 views
The document provides key monitoring tips for Apache Kafka, emphasizing the importance of monitoring broker health, message delivery, and capacity metrics. It suggests establishing a dashboard for quick health checks and monitoring specific issues like under-replicated partitions. Additionally, it highlights the significance of tracking performance, ensuring proper consumer behavior, and planning for capacity before reaching critical utilization levels.
Metrics are Not Enough: Monitoring Apache Kafka / Gwen Shapira (Confluent)
- 1. Gwen Shapira Product Manager @ Confluent And Apache Kafka Committer @gwenshap Metrics Are Not Enough Monitoring Apache Kafka and Streaming Applications
- 2. #1 Kafka Monitoring Tip: Please Monitor Apache Kafka.
- 3. Breaking tuning production databases since 1997 Apache Kafka PMC Product Manager for a monitoring product Tweeting a lot @gwenshap
- 4. Apache Kafka is a distributed system and has many components Controller
- 5. Things to keep an eye on • Broker health • Message delivery • Performance • Capacity
- 7. Kafka & Metrics Reporters • Pluggable interface • JMX reporter built in • TONS of metrics
- 8. TONS of Metrics • Broker throughput • Topic throughput • Disk utilization • Unclean leader elections • Network pool usage • Request pool usage • Request latencies – 30 request types, 5 phases each • Topic partition status counts: online, under replicated, offline • Log flush rates • ZK disconnects • Garbage collection pauses • Message delivery • Consumer groups reading from topics • …
- 9. #2 Kafka Monitoring Tip: Have a dashboard that lets you know “is everything ok?” in one look
- 10. Is the broker process up? • ps –ef | grep kafka • Do we receive metrics?
- 11. Under-replicated partitions • If you can monitor just one thing… • Is it a specific broker? • Cluster wide: • Out of resources • Imbalance • Broker: • Hardware • Noisy neighbor • Configuration • Garbage collection
- 12. Drill Down into Broker and Topic: Do we see a problem right here?
- 13. Check partition placement - is the issue specific to one broker?
- 14. #3 Kafka Monitoring Tip: Monitor Under-replicated Partitions
- 15. Canary • Produce an event • Try to consume the event 1s later • Did you get it?
- 16. Other important health metrics • Active Controller • ZK Disconnects • Unclean leader elections • ISR shrink/expand • # brokers
- 17. #3 Kafka Monitoring Tip: Don’t Watch the Dashboard
- 18. Few tips on broker health • Before 0.11.0.0 – restarting a broker is risky • Even after… • Before 1.0.0 – restarting a broker is slow • Especially with 5000+ partitions per broker Lesson: Only restart if you know why this will fix the issue.
- 19. Message Delivery
- 20. Delivery Guarantees • At most once • At least once • Exactly once • … and within N milliseconds
- 21. Are you meeting your guarantees, right now?
- 22. #4 Kafka Monitoring Tip: Monitoring brokers isn’t enough. You need to monitor events
- 23. Every Service that uses Kafka is a Distributed System Orders Service Stock Service Fulfilment Service Fraud Detection Service Mobile App Kafka
- 24. How to monitor? The infamous LinkedIn “Audit”: • Count messages when they are produced • Count messages when they are consumed • Check timestamps when they are consumed • Compare the results
- 25. Under Consumption • Reasons for under consumption: • Producers not handling errors and retried correctly • Misbehaving consumers, perhaps the consumer did not follow shutdown sequence • Real-time apps intentionally skipping messages
- 26. #5 Kafka Monitoring Life Tips: • producer.close(); • retries > 0 • Handle send() exceptions. • Use new consumer (0.9 and up)
- 27. Over Consumption • Reasons for over consumption • Consumers may be processing a set of messages more than once • Latency may be higher
- 28. Slow Consumers • Identify consumers and consumer groups that are not keeping up with data production • Compare a slow, lagging consumer (left) to a good consumer (right)
- 29. Other important client metrics • Producer retries • Producer errors • Consumer message lag – especially trends
- 31. #5 Kafka Monitoring Tip: Tune the parts that take time
- 32. Wrong: “OMG! Log Flush Time is 14ms” Is this high? How often we flush logs? Is it blocking? Who cares? Right: “OMG! We only process 50,000 requests per second. We need 10 times this” Why? IO threads are busy. Why? Waiting for flush. Why are we flushing so often? Log segments keep rotating. Lets configure larger segments!
- 34. Lifecycle of a request • Client sends request to broker • Network thread gets request and puts on queue • IO thread / handler picks up request and processes • Read / Write from/to local “disk” • Wait for other brokers to ack messages • Put response on queue • Network thread sends response to client
- 35. Produce and Fetch Request Latencies • Breakdown produce and fetch latencies through the entire request lifecycle • Each time segment correspond to a metric
- 36. How to make it faster? • Are you network, cpu, disk bound? • Do you need more threads? • Where do the threads spend their time?
- 37. Capacity Planning Key metrics that indicate a cluster is near capacity: • CPU • Network and thread pool usage • Request latencies • Network utilization • Disk utilization
- 38. #6 Kafka Monitoring Tip: By the time you reach 90% utilization. It is too late.
- 40. Few things to remember… • Alert on what’s important: Under-Replicated Partitions is a good start • DON’T JUST FIDDLE WITH STUFF • AND DON’T RESTART KAFKA FOR LOLS • If you don’t know what you are doing, it is ok. There’s support (and Cloud) for that.
- 41. Thank You! Find me on twitter: @gwenshap
Editor's Notes
- #3: Lets start with the obvious: If I ask you, “Hey, how is your Kafka cluster doing right now?” you should be able to tell me how many brokers are running and how many partitions are unavailable.
- #10: Lets start with the obvious: If I ask you, “Hey, how is your Kafka cluster doing right now?” you should be able to tell me how many brokers are running and how many partitions are unavailable.
- #11: Two ways to know if Kafka is up… check the process directly, or check if the other metrics you are getting are up to speed
- #15: Lets start with the obvious: If I ask you, “Hey, how is your Kafka cluster doing right now?” you should be able to tell me how many brokers are running and how many partitions are unavailable.
- #18: Lets start with the obvious: If I ask you, “Hey, how is your Kafka cluster doing right now?” you should be able to tell me how many brokers are running and how many partitions are unavailable.
- #22: Monitoring Kafka isn’t enough – Kafka is part of a system and producers and consumers are involved
- #23: Lets start with the obvious: If I ask you, “Hey, how is your Kafka cluster doing right now?” you should be able to tell me how many brokers are running and how many partitions are unavailable.
- #32: Lets start with the obvious: If I ask you, “Hey, how is your Kafka cluster doing right now?” you should be able to tell me how many brokers are running and how many partitions are unavailable.