Introduction to Apache Kafka
Download as PPTX, PDF2 likes8,247 views
Apache Kafka is an open-source distributed event streaming platform used for building real-time data pipelines and streaming apps. It was developed by LinkedIn in 2011 to solve problems with data integration and processing. Kafka uses a publish-subscribe messaging model and is designed to be fast, scalable, and durable. It allows both streaming and storage of data and acts as a central data backbone for large organizations.
Introduction to Apache Kafka
- 1. Introduction to Apache Kafka By: Dhaval Thumar WWW.AIMDEK.COM
- 2. Agenda • What is Apache Kafka? • How Kafka works? • History • Kafka Products • Kafka Core Components
- 3. What is Apache Kafka? ● Apache Kafka is a distributed streaming Open source platform developed by Apache Software Foundation, written in Scala and Java ○ A distributed system is a system with multiple components located on different machines that communicate and coordinate actions via messages in order to appear as a single system to the end-user. ○ Create real time data streams ○ Processing real time data stream
- 4. What is Apache Kafka?
- 5. How Kafka Works? • Kafka adopted Pub-Sub messaging architecture. Brocker Publisher Subscriber Message Broker Message Producer Message Consumer
- 6. History • Developed by Linkedin in 2011 to solve data integration problem
- 7. History
- 8. Kafka Products • Kafka Broker - Central server system • Kafka Client - Producer & Consumer Java & Scala Library • Kafka Connect - Data Integration tool • Kafka Stream - Java & Scala library for Real time stream processing application • KSQL
- 9. Kafka Core Concept • Producer • Consumer • Broker/Server • Cluster • Topic • Partition • Offset • Consumer Groups
- 10. Kafka Producer, Consumer, Broker
- 11. Kafka Topic Guys, I am collecting data from multiple producers. Which one do you want?
- 12. Kafka Topic Ok, give me the data sent by the smart- meter aproducer
- 13. Kafka Topic Well, the smart meter producer is sending three different types of data. 1. Current-load-sent every minute 2. Consumed-units-sent every hour 3. Input-current-fluctuations-sent as and when it happens Which one do you want?
- 14. Kafka Topic Well, give me the current-load Message
- 15. Kafka Topic Ok, so you are looking for a current-load message. That’s great! But we have a bunch of smart meter producers, and all of them are sending the current-load message. so which one of those are you interested in?
- 16. Kafka Topic
- 17. Kafka Topic Give me all the data that you received in the current load
- 18. Live Demo
- 20. www.aimdek.com AIMDek Technologies Pvt. Ltd. +91 78747 88766 | +1 844-744-4423 Sales: [email protected] General: [email protected] AIMDek Technologies Inc. +1 647-243-6116 Sales: [email protected] General: [email protected] AIMDek Technologies Inc. +1 647-243-6116 Sales: [email protected] General: [email protected] www.aimdek.com www.aimdek.com Canada India USA