SlideShare a Scribd company logo

What is Apache Kafka? Why is it so popular? Should I use it?

Guido Schmutz
Guido Schmutz

This document discusses Apache Kafka and provides an overview of its key properties and use cases. It notes that Kafka is a publish-subscribe messaging system that is horizontally scalable, highly available, durable, and schema-less. It can be used for streaming data integration and as a central data bus. The document outlines how Kafka fits into a complete streaming data architecture, including ingesting data from various sources, stream processing, batch integration, visualization, and connecting to data lakes, databases, and microservices. It positions Kafka as the central nervous system that streams can connect to and interact with.

Data & Analytics
1 of 25
Downloaded 71 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
BASEL | BERN | BRUGG | BUCHAREST | COPENHAGEN | DÜSSELDORF | FRANKFURT A.M. | FREIBURG I.BR. GENEVA | HAMBURG | LAUSANNE | MANNHEIM | MUNICH | STUTTGART | VIENNA | ZURICH http://guidoschmutz@wordpress.com@gschmutz What is Apache Kafka? Why is it so popular? Should I use it? Guido Schmutz Trivadis Speed Session 2019
BASEL | BERN | BRUGG | BUKAREST | DÜSSELDORF | FRANKFURT A.M. | FREIBURG I.BR. | GENF HAMBURG | KOPENHAGEN | LAUSANNE | MANNHEIM | MÜNCHEN | STUTTGART | WIEN | ZÜRICH Guido Working at Trivadis for more than 22 years Consultant, Trainer, Platform Architect for Java, Oracle, SOA and Big Data / Fast Data Oracle Groundbreaker Ambassador & Oracle ACE Director @gschmutz guidoschmutz.wordpress.com 174th edition
Event Hub Kafka Message Broker – Key properties • Publish / Subscribe Messaging – message can be consumed by 0 – n consumers • horizontally scalable – throughput increases with more nodes • highly available – no SPOF • durable – messages are not lost • Schema-less – Kafka broker has no knowledge on message content and format
Event Hub Vehicle Weather Streaming Data Sources
Event Hub Stream Data Integration Stream Data Integration Vehicle Weather Streaming Data Sources Streaming Data Ingestion
Event Hub Stream Data Integration Stream Data Integration Stream Analytics Vehicle Weather Streaming Data Sources Stream Analytics • Stream-to-Stream Joins • Stream-to-Table Joins • Time Windowed State Management • Event Pattern Detection • Machine Learning Model Execution (Inference)
Event Hub Stream Data Integration Stream Data Integration Stream Analytics Vehicle Weather Streaming Data Sources Data Lake / DWH Batch Data Integration Batch Visualize Data Lake Ingestion • Machine Learning • Graph Algorithms • Natural Language Processing
Event Hub Stream Data Integration Stream Data Integration Stream Analytics Vehicle Weather Legacy App Machine IIoT Batch Data Sources Streaming Data Sources Data Lake / DWH Batch Data Integration Batch Visualize
Event Hub Stream Data Integration Stream Data Integration Stream Analytics Vehicle Weather Legacy App Machine IIoT Batch Data Sources Streaming Data Sources Data Lake / DWH Batch Data Integration Batch Visualize
Event Hub Stream Data Integration Stream Data Integration Stream Analytics Vehicle Weather Legacy App Machine IIoT Stream Data Integration Batch Data Sources Streaming Data Sources CDC Data Lake / DWH Batch Data Integration Batch Visualize (Right-Time) Legacy Integration
Event Hub Stream Data Integration Stream Data Integration Stream Analytics Vehicle Weather Legacy App Machine IIoT Stream Data Integration Batch Data Sources Streaming Data Sources CDC Data Lake / DWH Batch Data Integration Batch Visualize (Right-Time) Legacy Integration
Event Hub Stream Data Integration Stream Data Integration Stream Analytics Vehicle Weather Legacy App Machine IIoT Stream Data Integration Batch Data Sources Streaming Data Sources CDC Stream Data Integration CDC Data Lake / DWH Batch Data Integration Batch Visualize (Right-Time) Legacy Integration
Event Hub Stream Data Integration Stream Data Integration Stream Analytics Vehicle Weather Legacy App Machine IIoT Stream Data Integration Batch Data Sources Streaming Data Sources CDC Stream Data Integration CDC Data Lake / DWH Batch Data Integration Batch Visualize (Right-Time) Legacy Integration
Event Hub Stream Data Integration Stream Data Integration Stream Analytics Vehicle Weather Legacy App Machine IIoT Stream Data Integration Batch Data Sources Streaming Data Sources CDC Stream Data Integration CDC Streaming Visualize Data Lake / DWH Batch Data Integration Stream Data Integration Batch Visualize Streaming Visualization
Event Hub Stream Data Integration Stream Data Integration Stream Analytics Vehicle Weather Legacy App Machine IIoT Stream Data Integration Batch Data Sources Streaming Data Sources CDC Stream Data Integration CDC Streaming Visualize Data Lake / DWH Batch Data Integration Stream Data Integration Stream Data Integration NOSQL NewSQL Batch Visualize Result Store Integration
Event Hub Stream Data Integration Stream Data Integration Stream Analytics Vehicle Weather Legacy App Machine IIoT Stream Data Integration Batch Data Sources Streaming Data Sources CDC Stream Data Integration CDC Streaming Visualize Data Lake / DWH Batch Data Integration Stream Data Integration Stream Data Integration NOSQL NewSQL Batch Visualize Micro service Highly Decoupled Modern Apps
Event Hub Stream Data Integration Stream Data Integration Stream Analytics Vehicle Weather Legacy App Machine IIoT Stream Data Integration Batch Data Sources Streaming Data Sources CDC Stream Data Integration CDC Streaming Visualize Data Lake / DWH Batch Data Integration Stream Data Integration Stream Data Integration NOSQL NewSQL Batch Visualize Micro service Micro service Highly Decoupled Modern Apps
Event Hub Stream Data Integration Stream Data Integration Stream Analytics Vehicle Weather Legacy App Machine IIoT Stream Data Integration Batch Data Sources Streaming Data Sources CDC Stream Data Integration CDC Streaming Visualize Data Lake / DWH Batch Data Integration Stream Data Integration Stream Data Integration NOSQL NewSQL Batch Visualize Micro service Micro service Gateway Data Source talks to Kafka through MQTT
Event Hub Stream Data Integration Stream Data Integration Stream Analytics Vehicle Weather Legacy App Machine IIoT Stream Data Integration Batch Data Sources Streaming Data Sources CDC Stream Data Integration CDC Streaming Visualize Data Lake / DWH Batch Data Integration Stream Data Integration Stream Data Integration NOSQL NewSQL Batch Visualize Micro service Micro service Gateway Kafka becomes central nervous system for data
Apache Kafka Kafka Cluster Consumer 1 Consume 2r Broker 1 Broker 2 Broker 3 Zookeeper Ensemble ZK 1 ZK 2ZK 3 Schema Registry Service 1 Management Control Center Kafka Manager KAdmin Producer 1 Producer 2 kafkacat Data Retention: • Never • Time (TTL) or Size-based • Log-Compacted based Producer3Producer3 ConsumerConsumer 3
• No SPoF, highly available • Consumer polls for new messages Apache Kafka • horizontally scalable, guaranteed order
Apache Kafka
Apache Kafka – Streaming Platform Source Connector Kafka Broker Sink Connector Stream Processing Schema Registry Kafka Kafka MQTT Proxy REST Proxy
Meine Präsentationen @ DOAG 2019 Location Intelligence: Real-Time Geofencing mit Kafka Mittwoch, 20. Nov., 14:00 - 14:45 Visualisierung von Streaming-Daten Donnerstag, 21. Nov., 12:00 - 12:45 Bi-direktionale Integration (Oracle) RDBMS und Apache Kafka Donnerstag, 21. Nov., 14:00 - 14:45
What is Apache Kafka? Why is it so popular? Should I use it?

More Related Content

What's hot (20)

PDF
Building Event-Driven (Micro) Services with Apache Kafka
Guido Schmutz
 
PDF
Kafka as an event store - is it good enough?
Guido Schmutz
 
PDF
Event Hub (i.e. Kafka) in Modern Data Architecture
Guido Schmutz
 
PDF
Building event-driven (Micro)Services with Apache Kafka
Guido Schmutz
 
PDF
Streaming Visualization
Guido Schmutz
 
PDF
Solutions for bi-directional integration between Oracle RDBMS & Apache Kafka
Guido Schmutz
 
PDF
Event Broker (Kafka) in a Modern Data Architecture
Guido Schmutz
 
PDF
Solutions for bi-directional Integration between Oracle RDMBS & Apache Kafka
Guido Schmutz
 
PDF
Building Event Driven (Micro)services with Apache Kafka
Guido Schmutz
 
PDF
Building Event Driven (Micro)services with Apache Kafka
Guido Schmutz
 
PDF
Building Event-Driven (Micro)Services with Apache Kafka
Guido Schmutz
 
PDF
Building event-driven (Micro)Services with Apache Kafka Ecosystem
Guido Schmutz
 
PDF
Fundamentals Big Data and AI Architecture
Guido Schmutz
 
PDF
Event Hub (i.e. Kafka) in Modern Data (Analytics) Architecture
Guido Schmutz
 
PDF
Microservices with Kafka Ecosystem
Guido Schmutz
 
PDF
Spark (Structured) Streaming vs. Kafka Streams
Guido Schmutz
 
PDF
Ingesting streaming data into Graph Database
Guido Schmutz
 
PDF
Solutions for bi-directional integration between Oracle RDBMS & Apache Kafka
Guido Schmutz
 
PDF
Streaming Visualization
Guido Schmutz
 
PDF
Location Analytics - Real-Time Geofencing using Kafka
Guido Schmutz
 
Building Event-Driven (Micro) Services with Apache Kafka
Guido Schmutz
 
Kafka as an event store - is it good enough?
Guido Schmutz
 
Event Hub (i.e. Kafka) in Modern Data Architecture
Guido Schmutz
 
Building event-driven (Micro)Services with Apache Kafka
Guido Schmutz
 
Streaming Visualization
Guido Schmutz
 
Solutions for bi-directional integration between Oracle RDBMS & Apache Kafka
Guido Schmutz
 
Event Broker (Kafka) in a Modern Data Architecture
Guido Schmutz
 
Solutions for bi-directional Integration between Oracle RDMBS & Apache Kafka
Guido Schmutz
 
Building Event Driven (Micro)services with Apache Kafka
Guido Schmutz
 
Building Event Driven (Micro)services with Apache Kafka
Guido Schmutz
 
Building Event-Driven (Micro)Services with Apache Kafka
Guido Schmutz
 
Building event-driven (Micro)Services with Apache Kafka Ecosystem
Guido Schmutz
 
Fundamentals Big Data and AI Architecture
Guido Schmutz
 
Event Hub (i.e. Kafka) in Modern Data (Analytics) Architecture
Guido Schmutz
 
Microservices with Kafka Ecosystem
Guido Schmutz
 
Spark (Structured) Streaming vs. Kafka Streams
Guido Schmutz
 
Ingesting streaming data into Graph Database
Guido Schmutz
 
Solutions for bi-directional integration between Oracle RDBMS & Apache Kafka
Guido Schmutz
 
Streaming Visualization
Guido Schmutz
 
Location Analytics - Real-Time Geofencing using Kafka
Guido Schmutz
 

Similar to What is Apache Kafka? Why is it so popular? Should I use it? (20)

PDF
Real-Time Analytics with Apache Cassandra and Apache Spark
Guido Schmutz
 
PDF
Real-Time Analytics with Apache Cassandra and Apache Spark,
Swiss Data Forum Swiss Data Forum
 
PDF
Kafka as your Data Lake - is it Feasible? (Guido Schmutz, Trivadis) Kafka Sum...
HostedbyConfluent
 
PDF
Kafka as your Data Lake - is it Feasible?
Guido Schmutz
 
PDF
Self-Service Data Ingestion Using NiFi, StreamSets & Kafka
Guido Schmutz
 
PPTX
Confluent-Ably-AWS-ID-2023 - GSlide.pptx
Ahmed791434
 
PDF
Fast Data – Fast Cars: Wie Apache Kafka die Datenwelt revolutioniert
confluent
 
PDF
Towards Data Operations
Andrea Monacchi
 
PPTX
Data & analytics challenges in a microservice architecture
Niels Naglé
 
PDF
Apache Kafka® Use Cases for Financial Services
confluent
 
PDF
Building end to end streaming application on Spark
datamantra
 
PPTX
Leveraging Azure Databricks to minimize time to insight by combining Batch an...
Microsoft Tech Community
 
PDF
Introduction to Streaming Analytics
Guido Schmutz
 
PDF
Architecting Applications With Multiple Open Source Big Data Technologies
Paul Brebner
 
PDF
Streaming Visualization
Guido Schmutz
 
PDF
A unified analytics platform with Kafka and Flink | Stephan Ewen, Ververica
HostedbyConfluent
 
PDF
Big Data Architecture
Guido Schmutz
 
PPTX
Big Data Analytics_basic introduction of Kafka.pptx
khareamit369
 
PDF
Data Pipelines with Spark & DataStax Enterprise
DataStax
 
PPTX
Revolutionize Your Data with Precisely and Confluent Streaming Technologies
Precisely
 
Real-Time Analytics with Apache Cassandra and Apache Spark
Guido Schmutz
 
Real-Time Analytics with Apache Cassandra and Apache Spark,
Swiss Data Forum Swiss Data Forum
 
Kafka as your Data Lake - is it Feasible? (Guido Schmutz, Trivadis) Kafka Sum...
HostedbyConfluent
 
Kafka as your Data Lake - is it Feasible?
Guido Schmutz
 
Self-Service Data Ingestion Using NiFi, StreamSets & Kafka
Guido Schmutz
 
Confluent-Ably-AWS-ID-2023 - GSlide.pptx
Ahmed791434
 
Fast Data – Fast Cars: Wie Apache Kafka die Datenwelt revolutioniert
confluent
 
Towards Data Operations
Andrea Monacchi
 
Data & analytics challenges in a microservice architecture
Niels Naglé
 
Apache Kafka® Use Cases for Financial Services
confluent
 
Building end to end streaming application on Spark
datamantra
 
Leveraging Azure Databricks to minimize time to insight by combining Batch an...
Microsoft Tech Community
 
Introduction to Streaming Analytics
Guido Schmutz
 
Architecting Applications With Multiple Open Source Big Data Technologies
Paul Brebner
 
Streaming Visualization
Guido Schmutz
 
A unified analytics platform with Kafka and Flink | Stephan Ewen, Ververica
HostedbyConfluent
 
Big Data Architecture
Guido Schmutz
 
Big Data Analytics_basic introduction of Kafka.pptx
khareamit369
 
Data Pipelines with Spark & DataStax Enterprise
DataStax
 
Revolutionize Your Data with Precisely and Confluent Streaming Technologies
Precisely
 
Ad

More from Guido Schmutz (11)

PDF
30 Minutes to the Analytics Platform with Infrastructure as Code
Guido Schmutz
 
PDF
Big Data, Data Lake, Fast Data - Dataserialiation-Formats
Guido Schmutz
 
PDF
ksqlDB - Stream Processing simplified!
Guido Schmutz
 
PDF
Location Analytics - Real-Time Geofencing using Apache Kafka
Guido Schmutz
 
PDF
Solutions for bi-directional integration between Oracle RDBMS and Apache Kafka
Guido Schmutz
 
PDF
Location Analytics Real-Time Geofencing using Kafka
Guido Schmutz
 
PDF
Location Analytics - Real Time Geofencing using Apache Kafka
Guido Schmutz
 
PDF
Stream Processing – Concepts and Frameworks
Guido Schmutz
 
PDF
Kafka as an Event Store - is it Good Enough?
Guido Schmutz
 
PDF
Solutions for bi-directional Integration between Oracle RDMBS & Apache Kafka
Guido Schmutz
 
PDF
Introduction to Stream Processing
Guido Schmutz
 
30 Minutes to the Analytics Platform with Infrastructure as Code
Guido Schmutz
 
Big Data, Data Lake, Fast Data - Dataserialiation-Formats
Guido Schmutz
 
ksqlDB - Stream Processing simplified!
Guido Schmutz
 
Location Analytics - Real-Time Geofencing using Apache Kafka
Guido Schmutz
 
Solutions for bi-directional integration between Oracle RDBMS and Apache Kafka
Guido Schmutz
 
Location Analytics Real-Time Geofencing using Kafka
Guido Schmutz
 
Location Analytics - Real Time Geofencing using Apache Kafka
Guido Schmutz
 
Stream Processing – Concepts and Frameworks
Guido Schmutz
 
Kafka as an Event Store - is it Good Enough?
Guido Schmutz
 
Solutions for bi-directional Integration between Oracle RDMBS & Apache Kafka
Guido Schmutz
 
Introduction to Stream Processing
Guido Schmutz
 
Ad

Recently uploaded (20)

PPTX
Feb 2021 Ransomware Recovery presentation.pptx
enginsayin1
 
PPTX
thid ppt defines the ich guridlens and gives the information about the ICH gu...
shaistabegum14
 
PDF
The European Business Wallet: Why It Matters and How It Powers the EUDI Ecosy...
Lal Chandran
 
PPTX
How to Add Columns and Rows in an R Data Frame
subhashenia
 
PDF
apidays Singapore 2025 - Surviving an interconnected world with API governanc...
apidays
 
PPTX
SHREYAS25 INTERN-I,II,III PPT (1).pptx pre
swapnilherage
 
PPTX
apidays Helsinki & North 2025 - From Chaos to Clarity: Designing (AI-Ready) A...
apidays
 
PDF
Data Science Course Certificate by Sigma Software University
Stepan Kalika
 
PDF
apidays Singapore 2025 - Streaming Lakehouse with Kafka, Flink and Iceberg by...
apidays
 
PDF
Research Methodology Overview Introduction
ayeshagul29594
 
PDF
OOPs with Java_unit2.pdf. sarthak bookkk
Sarthak964187
 
PDF
InformaticsPractices-MS - Google Docs.pdf
seshuashwin0829
 
PPTX
apidays Helsinki & North 2025 - Agentic AI: A Friend or Foe?, Merja Kajava (A...
apidays
 
PPTX
What Is Data Integration and Transformation?
subhashenia
 
PPTX
apidays Helsinki & North 2025 - Running a Successful API Program: Best Practi...
apidays
 
PDF
apidays Singapore 2025 - How APIs can make - or break - trust in your AI by S...
apidays
 
PPTX
apidays Singapore 2025 - Generative AI Landscape Building a Modern Data Strat...
apidays
 
PPTX
apidays Singapore 2025 - The Quest for the Greenest LLM , Jean Philippe Ehre...
apidays
 
PDF
A GraphRAG approach for Energy Efficiency Q&A
Marco Brambilla
 
PDF
apidays Singapore 2025 - Building a Federated Future, Alex Szomora (GSMA)
apidays
 
Feb 2021 Ransomware Recovery presentation.pptx
enginsayin1
 
thid ppt defines the ich guridlens and gives the information about the ICH gu...
shaistabegum14
 
The European Business Wallet: Why It Matters and How It Powers the EUDI Ecosy...
Lal Chandran
 
How to Add Columns and Rows in an R Data Frame
subhashenia
 
apidays Singapore 2025 - Surviving an interconnected world with API governanc...
apidays
 
SHREYAS25 INTERN-I,II,III PPT (1).pptx pre
swapnilherage
 
apidays Helsinki & North 2025 - From Chaos to Clarity: Designing (AI-Ready) A...
apidays
 
Data Science Course Certificate by Sigma Software University
Stepan Kalika
 
apidays Singapore 2025 - Streaming Lakehouse with Kafka, Flink and Iceberg by...
apidays
 
Research Methodology Overview Introduction
ayeshagul29594
 
OOPs with Java_unit2.pdf. sarthak bookkk
Sarthak964187
 
InformaticsPractices-MS - Google Docs.pdf
seshuashwin0829
 
apidays Helsinki & North 2025 - Agentic AI: A Friend or Foe?, Merja Kajava (A...
apidays
 
What Is Data Integration and Transformation?
subhashenia
 
apidays Helsinki & North 2025 - Running a Successful API Program: Best Practi...
apidays
 
apidays Singapore 2025 - How APIs can make - or break - trust in your AI by S...
apidays
 
apidays Singapore 2025 - Generative AI Landscape Building a Modern Data Strat...
apidays
 
apidays Singapore 2025 - The Quest for the Greenest LLM , Jean Philippe Ehre...
apidays
 
A GraphRAG approach for Energy Efficiency Q&A
Marco Brambilla
 
apidays Singapore 2025 - Building a Federated Future, Alex Szomora (GSMA)
apidays
 

What is Apache Kafka? Why is it so popular? Should I use it?

  • 1. BASEL | BERN | BRUGG | BUCHAREST | COPENHAGEN | DÜSSELDORF | FRANKFURT A.M. | FREIBURG I.BR. GENEVA | HAMBURG | LAUSANNE | MANNHEIM | MUNICH | STUTTGART | VIENNA | ZURICH http://[email protected]@gschmutz What is Apache Kafka? Why is it so popular? Should I use it? Guido Schmutz Trivadis Speed Session 2019
  • 2. BASEL | BERN | BRUGG | BUKAREST | DÜSSELDORF | FRANKFURT A.M. | FREIBURG I.BR. | GENF HAMBURG | KOPENHAGEN | LAUSANNE | MANNHEIM | MÜNCHEN | STUTTGART | WIEN | ZÜRICH Guido Working at Trivadis for more than 22 years Consultant, Trainer, Platform Architect for Java, Oracle, SOA and Big Data / Fast Data Oracle Groundbreaker Ambassador & Oracle ACE Director @gschmutz guidoschmutz.wordpress.com 174th edition
  • 3. Event Hub Kafka Message Broker – Key properties • Publish / Subscribe Messaging – message can be consumed by 0 – n consumers • horizontally scalable – throughput increases with more nodes • highly available – no SPOF • durable – messages are not lost • Schema-less – Kafka broker has no knowledge on message content and format
  • 5. Event Hub Stream Data Integration Stream Data Integration Vehicle Weather Streaming Data Sources Streaming Data Ingestion
  • 6. Event Hub Stream Data Integration Stream Data Integration Stream Analytics Vehicle Weather Streaming Data Sources Stream Analytics • Stream-to-Stream Joins • Stream-to-Table Joins • Time Windowed State Management • Event Pattern Detection • Machine Learning Model Execution (Inference)
  • 7. Event Hub Stream Data Integration Stream Data Integration Stream Analytics Vehicle Weather Streaming Data Sources Data Lake / DWH Batch Data Integration Batch Visualize Data Lake Ingestion • Machine Learning • Graph Algorithms • Natural Language Processing
  • 8. Event Hub Stream Data Integration Stream Data Integration Stream Analytics Vehicle Weather Legacy App Machine IIoT Batch Data Sources Streaming Data Sources Data Lake / DWH Batch Data Integration Batch Visualize
  • 9. Event Hub Stream Data Integration Stream Data Integration Stream Analytics Vehicle Weather Legacy App Machine IIoT Batch Data Sources Streaming Data Sources Data Lake / DWH Batch Data Integration Batch Visualize
  • 10. Event Hub Stream Data Integration Stream Data Integration Stream Analytics Vehicle Weather Legacy App Machine IIoT Stream Data Integration Batch Data Sources Streaming Data Sources CDC Data Lake / DWH Batch Data Integration Batch Visualize (Right-Time) Legacy Integration
  • 11. Event Hub Stream Data Integration Stream Data Integration Stream Analytics Vehicle Weather Legacy App Machine IIoT Stream Data Integration Batch Data Sources Streaming Data Sources CDC Data Lake / DWH Batch Data Integration Batch Visualize (Right-Time) Legacy Integration
  • 12. Event Hub Stream Data Integration Stream Data Integration Stream Analytics Vehicle Weather Legacy App Machine IIoT Stream Data Integration Batch Data Sources Streaming Data Sources CDC Stream Data Integration CDC Data Lake / DWH Batch Data Integration Batch Visualize (Right-Time) Legacy Integration
  • 13. Event Hub Stream Data Integration Stream Data Integration Stream Analytics Vehicle Weather Legacy App Machine IIoT Stream Data Integration Batch Data Sources Streaming Data Sources CDC Stream Data Integration CDC Data Lake / DWH Batch Data Integration Batch Visualize (Right-Time) Legacy Integration
  • 14. Event Hub Stream Data Integration Stream Data Integration Stream Analytics Vehicle Weather Legacy App Machine IIoT Stream Data Integration Batch Data Sources Streaming Data Sources CDC Stream Data Integration CDC Streaming Visualize Data Lake / DWH Batch Data Integration Stream Data Integration Batch Visualize Streaming Visualization
  • 15. Event Hub Stream Data Integration Stream Data Integration Stream Analytics Vehicle Weather Legacy App Machine IIoT Stream Data Integration Batch Data Sources Streaming Data Sources CDC Stream Data Integration CDC Streaming Visualize Data Lake / DWH Batch Data Integration Stream Data Integration Stream Data Integration NOSQL NewSQL Batch Visualize Result Store Integration
  • 16. Event Hub Stream Data Integration Stream Data Integration Stream Analytics Vehicle Weather Legacy App Machine IIoT Stream Data Integration Batch Data Sources Streaming Data Sources CDC Stream Data Integration CDC Streaming Visualize Data Lake / DWH Batch Data Integration Stream Data Integration Stream Data Integration NOSQL NewSQL Batch Visualize Micro service Highly Decoupled Modern Apps
  • 17. Event Hub Stream Data Integration Stream Data Integration Stream Analytics Vehicle Weather Legacy App Machine IIoT Stream Data Integration Batch Data Sources Streaming Data Sources CDC Stream Data Integration CDC Streaming Visualize Data Lake / DWH Batch Data Integration Stream Data Integration Stream Data Integration NOSQL NewSQL Batch Visualize Micro service Micro service Highly Decoupled Modern Apps
  • 18. Event Hub Stream Data Integration Stream Data Integration Stream Analytics Vehicle Weather Legacy App Machine IIoT Stream Data Integration Batch Data Sources Streaming Data Sources CDC Stream Data Integration CDC Streaming Visualize Data Lake / DWH Batch Data Integration Stream Data Integration Stream Data Integration NOSQL NewSQL Batch Visualize Micro service Micro service Gateway Data Source talks to Kafka through MQTT
  • 19. Event Hub Stream Data Integration Stream Data Integration Stream Analytics Vehicle Weather Legacy App Machine IIoT Stream Data Integration Batch Data Sources Streaming Data Sources CDC Stream Data Integration CDC Streaming Visualize Data Lake / DWH Batch Data Integration Stream Data Integration Stream Data Integration NOSQL NewSQL Batch Visualize Micro service Micro service Gateway Kafka becomes central nervous system for data
  • 20. Apache Kafka Kafka Cluster Consumer 1 Consume 2r Broker 1 Broker 2 Broker 3 Zookeeper Ensemble ZK 1 ZK 2ZK 3 Schema Registry Service 1 Management Control Center Kafka Manager KAdmin Producer 1 Producer 2 kafkacat Data Retention: • Never • Time (TTL) or Size-based • Log-Compacted based Producer3Producer3 ConsumerConsumer 3
  • 21. • No SPoF, highly available • Consumer polls for new messages Apache Kafka • horizontally scalable, guaranteed order
  • 23. Apache Kafka – Streaming Platform Source Connector Kafka Broker Sink Connector Stream Processing Schema Registry Kafka Kafka MQTT Proxy REST Proxy
  • 24. Meine Präsentationen @ DOAG 2019 Location Intelligence: Real-Time Geofencing mit Kafka Mittwoch, 20. Nov., 14:00 - 14:45 Visualisierung von Streaming-Daten Donnerstag, 21. Nov., 12:00 - 12:45 Bi-direktionale Integration (Oracle) RDBMS und Apache Kafka Donnerstag, 21. Nov., 14:00 - 14:45