kafka simplicity and complexity
Download as PPTX, PDF1 like98 views
Agile Lab is an Italian company that specializes in leveraging innovative technologies like machine learning, big data, and artificial intelligence to satisfy customers' objectives. They have over 50 specialists with deep experience in production environments. The company believes in investing in its team through conferences, R&D projects, and welfare benefits. They also release open source frameworks on GitHub and share knowledge through meetups in Milan and Turin.
kafka simplicity and complexity
- 3. Agile is our philosophy to transform business ideas in real use cases. We understand rapidly the key elements of the Customer’s challenge, without fear for any challenge. We are an efficient organization totally focused on leveraging innovative technologies to satisfy the customer’s objectives and time-to-market. Our reference handbook for organization is public: https://publicagilefactory.gitlab.io/handbook/ The secret of getting ahead is getting started - Mark Twain www.agilelab.it
- 4. More than 50 specialists, who have a deep understanding of the real production environments. We constantly invest time and effort to increase our skills on new technologies and innovative analysis methodologies. International Conferences, R&D Projects, Smart Working, Welfare, Work for Equity… We believe in our team and we invest for it. Our reference handbook for organization is public: https://publicagilefactory.gitlab.io/handbook / www.agilelab.it OurTeam Think about the future, think about what you could do, don’t fear anything - Rita Levi Montalcini
- 5. www.agilelab.it In memory and Machine learning Hadoop Frameworks and storage format Change Data Capture NoSQL Data Ingestion and Hi-scalable applications Deployment and resource management Cloud PaaS deployment Deep Learning Development and Data Visualization Certified
- 6. CHECKOUTOUR FRAMEWORKONGITHUB www.agilelab.it Check ourframeworksreleasedfor“native” DataQualityinBig Dataenvironmentsorour SparkSearchlibrary. SHARING KNOWLEDGE IN MEETUPS Don’tmissourMeetUps appointmentsbothin Milanoand Torino. Darwin Schemaregistry WASP isourframeworkforstreaming analyticsatscale.Ithasbeenreleased OpenSourcesince thebeginningandis goingtobe evolvedwithnew developments.
- 7. Kafka
- 8. www.agilelab.it • General purpose Message/Event Bus • Real-Time Stream Processing in collaboration with other tools • Collecting User Activity Data • Collecting Operational Metrics from applications, server or devices • Log Aggregation • Change Data Capture • Commit Log for distributed systems
- 9. www.agilelab.it • As distributed system and services increasingly become part of modern architecture, this makes for a fragile system • As integration complexity increases, evolvability decreases Slave System ETL ETL Master System Reporting System Service 1 Service 3 Service 2 ETL
- 10. www.agilelab.it Decoupling integration patterns increase evolvability !!! Slave System Reporting System Service 1 Service 3Service 2 Kafka Kafka is not a «master» system, but a «central» system for service and data integration: real time and event based
- 11. www.agilelab.it Focuson: • Simplicity • Throughput • ExtremeScalability Not for: • Large batch exports • Low Latency
- 12. • Message ordering is only guaranteed within a partition for a topic • Scaling is achieved through partitions • At least once delivery • For a topic with replication factor N, Kafka can tolerate up to N-1 server failures without “losing” any messages committed to the log • Messages sent by a producer to a particular topic partition will be appended in the order they are sent • A consumer instance sees messages in the order they are store in the log
- 13. • Topics are broken up into ordered commit logs called partitions. • To each message in a partition a sequential id (called offset) is assigned • Data is retained for a configurable period of time
- 14. Producer
- 15. • Producers publish to a topic of their choosing (push) • Load can be distributed • Typically by “round-robin” • Can also do “semantic partitioning” based on a key in the message • Brokers load balance by partition • Can support async (less durable) sending • All nodes can answer metadata request about • Which server is alive • Where leaders are for the partition of a topic
- 16. Consumer
- 17. • Multiple Consumers can read from the same topic • Each Consumer is responsible for managing it’s own offset • Messages stay on Kafka… they are not removed after they are consumed • Consumers can go away • And then came back
- 19. Why is Kafka so scalable?
- 20. @Linkedin: 100+ clusters 4K brokers 100K topics 7M partitions 81M msg/sec ???!!! On Prem scenario: 6 x Server: • Xeon 2.5 GHz – 6 cores • 6 x HDD 7200 rpm • 32 GB RAM • 1Gb Ethernet 1 topic – 6 partitions – 3x replica – 1 producer: • 800K msg/sec • 133K msg/sec x node 1 consumer from a 3x replicated topic: • 950K msg/sec Latency end to end: 3ms 99th 14ms 99.9th Reference: KV DB with persistence 10K write/sec x node KV DB in memory 50K write/sec x node Kafka is persistent...so what ??
- 21. Where is the trick?
- 22. Incredible complex architecture from ‘90? (do you know this system?)
- 23. Why is so fast?
- 24. It is simple !!!
- 25. • Focus on sequential I/O no Random I/O • How is this possible with multiple writers and readers ? • Page cache • No Indexes • A partition it’s just a file in append mode • ZeroCopy: directly copy data from pagecache to NIC buffer through OS sendfile system call, skipping userspace and socket buffer copies Writes are sequential by definition Page Cache – no disk access
- 27. • Everything in Kafka is batched and compressed !!! • Communication is done via a high performance binary API over TCP protocol Fake !!!! • Batching of data to reduce network calls, and also converting a lot of random writes into sequential ones. • Compression of batches (and not individual messages) using LZ4, SNAPPY or GZIP codecs.
- 28. Questions ?
- 29. We are hiring www.agilelab.it [email protected] www.linkedin.com/company/agile-lab Thank you!