Apache Kafka - Overview
Download as PPTX, PDF1 like536 views
The document discusses the need for a messaging system, specifically focusing on Apache Kafka and its components such as producers, brokers, and consumers. It highlights Kafka's advantages, including high throughput, scalability, durability, and fault tolerance, as well as its ability to decouple data pipelines. Additionally, the document outlines Kafka's architecture and its applications in real-time streaming and data management systems.
Apache Kafka - Overview
- 1. From : Madhusudhana IG Group Date :14-April-2018
- 2. Need of Messaging system What , Why , How of Apache Kafka Producers, Brokers , Consumers ,Topics and Partitions Developing Producers and Consumers Questions Overview
- 3. Need of a messaging system
- 4. Front end UVData pipe line It often starts with one data pipe line
- 5. Front end UV Ledger systems Instrument management system Reuse of existing providers for new consumers
- 6. Front end Ledger systems Instrument management system Security systems Database server Data ware house Account Maintenance system Real time monitoring Overnight chargesPricing system Eventually the solution becomes the problem
- 7. Front end Ledger systems Instrument management system Security systems Database server Data ware house Account Maintenance system Real time monitoring Overnight chargesPricing system Messaging system
- 8. How Kafka solves this problem
- 9. Kafka decouples data pipe lines Data warehouse Account Maintenance system Ledger system Kafka Real time monitoring Overnight charges Pricing system Instrument management system Open position system
- 10. What is Apache Kafka? Apache Kafka is a distributed publish-subscribe message system It was originally developed at LinkedIn, later open sourced in 2011 and became a part of Apache project Kafka is fast , scalable ,durable , fault-tolerant and distributed by design
- 11. Kafka Features High Throughput: Provides support for Hundreds of thousand of messages Scalability: Highly scalable distributed systems with no downtime Durability: Provides support to persisting messages on disk Replication: Messages can be replicated across clusters , witch supports multiple subscribers Stream Processing: Kafka can be used along with real time streaming applications like Spark and storm
- 13. Kafka Cluster Producer Consumer Producer ConsumerKafka cluster Broker 1 Broker 3 Broker 2 Zookeeper
- 17. Apache Kafka architecture 0 Topic Partition1 Partition2 Partition3 Producer A Consumer A Consumer B Consumer C Broker1 0 1 2 10 1
- 18. Producer and consumer DEMO