From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 Hours to 30 Minutes
0 likes130 views
The document discusses improving Kafka Connect source connector ingestion from 7 hours to just 30 minutes through increased batch sizes. It highlights issues with inefficiency and presents a solution by integrating JMX metrics with monitoring tools like Prometheus and Grafana. Testing results demonstrate significant reductions in processing time and improved throughput by adjusting the batch size.
From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 Hours to 30 Minutes
- 1. From the Trenches: Improving Kafka Connect Source Connector Ingestion from 7 Hours to 30 Minutes
- 2. Improving Kafka Connect Ingestion K af k a Summi t Lond on 20 24 – Raf ae l N at al i
- 3. Kafka Summit London 2024 Rafael Natali /rafaelnatali @rafaelmnatali marionete.co.uk
- 4. PROBLEMS
- 7. MONITORING DOCUMENTATION Enable JMX Metrics Integrate Prometheus + Grafana Overall view of the Kafka Connect https://www.confluent.io/en-gb/blog/how-to-increase-throughput-on-kafka-connect-source-connectors/
- 12. TESTING
- 13. BATCH.SIZEINCREASE batch.size = number of records * record size average in bytes "producer.override.batch.size": 739500 batch.size = 1500 * 493 bytes batch.size = 739500 bytes
- 14. RESULTS
- 18. SUMMARY 7h 30min <16kb 600Kb 35 1500