Kafka - Linkedin's messaging backbone
Download as PPTX, PDF3 likes857 views
Kafka is a high-throughput distributed messaging system developed by LinkedIn, designed to handle real-time data feeds and stream processing with features like low latency and fault tolerance. It supports extensive messaging needs for over 1,000 companies, including major firms like Netflix and Uber, and processes over 1.3 trillion messages daily, showcasing its scalability and performance. At LinkedIn, Kafka is utilized for various applications, such as monitoring, user tracking, and notification systems, while operating on a robust infrastructure with specific hardware configurations.
Kafka - Linkedin's messaging backbone
- 1. Kafka - Linkedin’s messaging backbone
- 3. Who are we ? ▪ Kafka SRE at LinkedIn ▪ Site Reliability Engineering – Administrators – Architects – Developers ▪ Keep the site running, always
- 4. Presenters ▪ Clark Haskins – Manager for Data Infra Streaming SRE (Mountain View, CA) ▪ Ayyappadas Ravindran – Staff Site Reliability Engineer, Data Infra Streaming (Bengaluru) ▪ Akash Vacher ▪ Site Reliability Engineer, Data Infra Streaming (Bengaluru)
- 5. Agenda ▪ What the heck is Kafka ? – Brief intro – Motivation to build Kafka ▪ Okay, why should I bother ? – Kafka facts, scale & performance ▪ You have my attention, tell me more ! – Core concepts – Operating kafka – Kafka @ Linkedin ▪ Nice, where all do you use kafka ? – Tale of two applications ▪ Have questions ?
- 6. What the heck is Kafka? ▪ A high-throughput distributed messaging system ▪ Developed at Linkedin and open sourced in early 2011 ▪ Implemented in Scala and Java ▪ Linkedin’s messaging backbone ▪ Kafka powers around 1000 companies including Linkedin, Yahoo!, Netflix, Uber, Twitter and many more If data is lifeblood of high technology, Apache Kafka is the circulatory system used in Linkedin – Todd Palino (Staff SRE Engineer Linkedin)
- 7. Motivation to create Kafka ? ▪ Needed a unified platform to handle all real time data feeds and stream processing ▪ Wanted a messaging system with high throughput to support high volume event feeds ▪ Needed data persistence for offline systems and in case of service recovery ▪ Low latency ▪ Fault tolerant ▪ Linearly scalable Okay ! what was the motivation to create Kafka?
- 8. Before
- 9. After
- 10. How is Kafka used at Linkedin? ▪ Application and System Monitoring (inGraphs) ▪ User tracking on Linkedin web sites ▪ Email, push & SMS notifications ▪ Live search updates ▪ Samza Jobs (standardization, call graph and more) ▪ Database Replication
- 11. Okay, why should I bother? ▪ Over 1,300,000,000,000 messages are transported via Kafka every day at LinkedIn ▪ 300 Terabytes of inbound and 900 Terabytes of outbound traffic ▪ 4.5 Million messages per second, on single cluster ▪ Kafka runs on around 1300 servers at LinkedIn hmmm .. ! How good is Kafka ?
- 12. You have my attention, tell me more ! ▪ Building blocks – Message – Producers – Consumers – Topics – Partitions – Segments – Brokers – Replicas Awesome !! I am in Tell me more !
- 13. Bird’s eye view
- 14. The data continues ..
- 16. Performance recipes ▪ OS page cache ▪ Linear IO, never fear the file system ! ▪ sendfile(), system call ▪ Message batching Dude, tell me the performance secret!!
- 17. Operating Kafka ▪ Broker Hardware – Cisco C240, Intel xeon, 64GB RAM , 14 disk Raid-10 ▪ Zookeeper Hardware – 5 + 1 ensemble, 64GB RAM, 500GB SSD ▪ Monitoring – Lag monitoring – Under Replicated Partitions – Unclean leader election – Burrow ▪ Cluster rebalance – Sizewise rebalance – Partitionwise rebalance Tell me how you manage this beast !
- 18. Mirror Maker and Audit
- 21. Kafka Audit(data transport time)
- 22. Kafka @ Linkedin ▪ Cluster Types – Tracking – Metrics – Queuing ▪ Kafka Rest ▪ Schema Registry
- 23. Kafka @ Linkedin - Schema registry
- 24. Autometrics ▪ Building Blocks – Sensors – EventBus – Kafka Rest – Kafka cluster – Kafka consumer – RRD – Front end ▪ Facts & Figures – 320,000,000 metrics collected per minute ▪ 530 TB of disk space ▪ Over 210,000 metrics collected per service
- 25. InGraphs
- 26. Kafka for database replication - Master slave
- 27. Kafka for database replication - Multi master
- 28. Have questions?