What’s Slowing Down Your Kafka Pipeline? With Ruizhe Cheng and Pete Stevenson | Current 2022
0 likes471 views
The document discusses the challenges of monitoring Kafka pipelines and introduces eBPF as a no-effort approach to enhance Kafka observability. It explains the use of Pixie's eBPF capabilities for network tracing and CPU profiling to identify performance bottlenecks. Key features highlighted include automatic symbol generation, support for various compiled languages, and the ability to monitor critical Kafka metrics in a Kubernetes environment.
![Eager rebalancing protocol
Stops the world!
Kafka rebalancing Protocols
[1] https://www.confluent.io/blog/cooperative-rebalancing-in-kafka-streams-consumer-ksqldb/
Incremental cooperative rebalancing protocol
Stops only some consumers as needed](https://image.slidesharecdn.com/ruizhechengpetestevensonbr05510-221023134028-6ffd681a/85/What-s-Slowing-Down-Your-Kafka-Pipeline-With-Ruizhe-Cheng-and-Pete-Stevenson-Current-2022-18-320.jpg)
What’s Slowing Down Your Kafka Pipeline? With Ruizhe Cheng and Pete Stevenson | Current 2022
- 1. What’s slowing down your Kafka pipeline? A no-effort approach using eBPF October 5, 2022 Ruizhe (Ryan) Cheng, John P. Stevenson A CNCF sandbox project
- 2. Ruizhe (Ryan) Cheng Software Engineer at New Relic (Pixie team) Hi! 👋
- 3. Kafka as the centerpiece
- 5. Kafka Observability CPU, Memory, Network IO etc.
- 6. Kafka Observability CPU, Memory, Network IO etc. Pods, Nodes, Deployments etc.
- 7. Kafka Observability CPU, Memory, Network IO etc. Pods, Nodes, Deployments etc. Threads, Memory, GC etc.
- 8. Kafka Observability CPU, Memory, Network IO etc. Pods, Nodes, Deployments etc. Threads, Memory, GC etc. Topics, partitions, offsets etc.
- 9. Kafka Observability ● JMX: Java Management extension ● Agent collects data from JMX ● Data exported for visualization and storage
- 10. Monitoring needed on ● Producers ● Consumers ● Brokers Kafka Observability
- 11. Agenda ● What is eBPF and Why monitor Kafka with eBPF ● Monitor Kafka with Pixie’s eBPF network tracer ● Find CPU bottlenecks in Kafka with Pixie’s CPU profiler
- 12. What is eBPF? An Analogy An eBPF probe is like a breakpoint in a debugger ● Interrupts execution when breakpoint is reached ● But, unlike a breakpoint, a small eBPF program runs ● Decouples your application and the monitoring software
- 13. an open source, eBPF based, observability platform. What Pixie provides: ● Metrics: CPU, memory, network, JVM stats, etc. ● Network tracing: Kafka, HTTP, MySQL, etc. ○ No instrumentation ● CPU profiling (flamegraphs) ○ Always on
- 14. Monitor Kafka using Pixie’s network traffic tracer
- 15. Demo
- 16. Demo: Pixie Kafka Overview
- 17. Kafka Rebalances
- 18. Eager rebalancing protocol Stops the world! Kafka rebalancing Protocols [1] https://www.confluent.io/blog/cooperative-rebalancing-in-kafka-streams-consumer-ksqldb/ Incremental cooperative rebalancing protocol Stops only some consumers as needed
- 19. Kafka Consumer Lag Measured in ● partition offset ● Wall clock time
- 20. Demo: Consumer rebalances + Consumer lag
- 21. Find CPU bottlenecks in Kafka with Pixie’s CPU profiler
- 22. John P. (Pete) Stevenson Software Engineer at New Relic (Pixie team) Hi! 👋
- 25. Flamegraphs: width (on x-axis) represents time. Processor::run() sits in a “wide bar.” Our program spends more time here. ThreadPoolExecutor::runWorker() sits in a “narrow bar.” Using less time here.
- 26. Flamegraphs: stack traces (on y-axis) show what was happening. An example stack trace from the Kafka broker. run() called poll() ... which called pollSelectionKeys() ... which called write() ... which called writev0() ... which called into libc, ... which called into the kernel.
- 27. Flamegraphs: stack traces (on y-axis) show what was happening. An example stack trace from the Kafka broker. run() called poll() ... which called pollSelectionKeys() ... which called write() ... which called IOUtil.write() ... which called writev0() ... which called into libc, ... which called into the kernel.
- 28. Flamegraphs: stack traces (on y-axis) show what was happening. An example stack trace from the Kafka broker. run() called poll() ... which called pollSelectionKeys() ... which called write() ... which called IOUtil.write() ... which called writev0() ... which called into libc, ... which called into the kernel.
- 29. Flamegraphs: stack traces (on y-axis) show what was happening. An example stack trace from the Kafka broker. run() called poll() ... which called pollSelectionKeys() ... which called write() ... which called IOUtil.write() ... which called writev0() ... which called into libc, ... which called into the kernel.
- 30. An example stack trace from the Kafka broer. run() called poll() ... which called pollSelectionKeys() ... which called write() ... which called IOUtil.write() ... which called writev0() ... which called into libc, ... which called into the kernel. This gap above /lib/ld-musl-x86_64.so, ... this little blank space, is time attributed to libc. System call overhead.
- 31. Java profiling landscape Native - Runs in the JVM. - Native access to code symbols. - Safepoint bias issue. External - Uses Linux kernel for sampling. - Symbols not directly available. - No safepoint bias; provides full system context.
- 32. Java profiling landscape Native - Runs in the JVM. - Native access to code symbols. - Safepoint bias issue. External - Uses Linux kernel for sampling. - Symbols not directly available. - No safepoint bias; provides full system context. May be difficult to setup & use: ● App. redeploy ● Hard to exfiltrate profiling data from prod. Stack traces are not useful without symbols!
- 33. Fully automated, always on, in prod. ● No app. redeploy (just start the Pixie daemonset). ● Supports compiled languages (C/C++/Go/Rust). ● External profiler (eBPF). ● Automatic Java symbols (using JVMTI). For Kubernetes, Pixie to the rescue
- 34. Kafka Data Compression Compression Type Compression Ratio CPU Usage Network / Disk Gzip High High Low Snappy Medium Moderate Medium Zstd Medium Moderate Medium Lz4 Low Low High
- 35. Demo: Broker data compression
- 36. Gzip: 26.8% CPU Lz4: 7.7% CPU
- 37. Conclusion Kafka monitoring is challenging Important Kafka metrics: ○ _Consumer rebalance events ○ _Consumer lag ○ _Data compression Pixie provides eBPF based observability on K8s, ○ _that is fully decoupled and automated, ○ _including Kafka protocol tracing and parsing, ○ _including CPU performance flamegraphs, ○ _with symbols for Java automatically populated