Apache Kafka and MQTT - Overview, Comparison, Use Cases, Architectures
1 like17,625 views
The document discusses the integration of manufacturing 4.0, connected vehicles, and smart city solutions through the use of MQTT and Apache Kafka technologies. It outlines industry trends, application examples from companies like Daimler and ZF, and presents various reference architectures for enhancing operational efficiency and real-time data processing across sectors. Additionally, it highlights solutions provided by HiveMQ and Confluent for managing high-volume data streaming and connectivity in these domains.
Apache Kafka and MQTT - Overview, Comparison, Use Cases, Architectures
- 1. Manufacturing 4.0, Connected Vehicles, Mobility Services, and Smart City Real Time Data Processing and Analytics powered by Apache Kafka and MQTT Kai Waehner Field CTO Confluent [email protected] linkedin.com/in/kaiwaehner @KaiWaehner www.confluent.io www.kai-waehner.de Dominik Obermaier CTO HiveMQ [email protected] www.linkedin.com/in/dobermai @dobermai www.hivemq.com 1
- 2. Agenda 1. Industry Trends - Connected World, Industry 4.0 2. Manufacturing 3. Connected Vehicles 4. Mobility Services 5. Smart City 6. HiveMQ and Confluent
- 3. Agenda 1. Industry Trends - Connected World, Industry 4.0 2. Manufacturing 3. Connected Vehicles 4. Mobility Services 5. Smart City 6. HiveMQ and Confluent
- 4. Industry Trends - Connected World 4 People on the Internet
- 5. A Streaming Platform is the Underpinning of an Event-driven Architecture Ubiquitous connectivity Globally scalable platform for all event producers and consumers Immediate data access Data accessible to all consumers in real time Single system of record Persistent storage to enable reprocessing of past events Continuous queries Stream processing capabilities for in-line data transformation Microservices DBs SaaS apps Mobile Customer 360 Real-time fraud detection Data warehouse Producers Consumers Database change Microservices events SaaS data Customer experience s Streams of real time events Stream processing apps
- 6. Data Movement from Device to Enterprise Services SCADA ERP Enterprise Data Center • MQTT Protocol decouples data integration bindings • Bi-directional data movement between devices and data center • Support for unreliable network MQTT Broker ...
- 7. Agenda 1. Industry Trends - Connected World, Industry 4.0 2. Manufacturing 3. Connected Vehicles 4. Mobility Services 5. Smart City 6. HiveMQ and Confluent
- 8. Food Value Chain IoT-Based and Data-Driven Single source of truth across the food value chain (in the factories, and across regions) Business critical operations (tracking, calculations, alerts, …)
- 9. Daimler - Automotive Manufacturing Vehicle Diagnostic System (VDS) • Critical part of manufacturing process that cannot go down HiveMQ Solution • Efficient way to share information between test devices on factory floor and enterprise IT systems • Test devices behave properly if network connection is dropped and reconnected Results ● Rolled out to 24 factories around the world ● 10,000 testing devices connected ● Generating 470 million messages/month bit.ly/hivemq-daimler
- 10. Reference Architecture - Manufacturing
- 11. Agenda 1. Industry Trends - Connected World, Industry 4.0 2. Manufacturing 3. Connected Vehicles 4. Mobility Services 5. Smart City 6. HiveMQ and Confluent
- 12. ‘My Porsche’ A digital service platform for customers, fans, and enthusiasts https://medium.com/porschedev
- 13. Autonomic - Connecting Millions of Cars Transportation Mobility Cloud (TMC) = Accessing connected car data and sending remote commands as the foundation for mobility applications: driver safety, preventive maintenance, fleet management… HiveMQ Solution ● High degree of customization ● Scale to millions of connections ● Easy to create a proof of concept Results ● Millions of cars connected via TMC ● Extensive use of HiveMQ Extension SDK ○ forward all MQTT messages to Kafka ○ messages distributed to different microservices & applications bit.ly/hivemq-autonomic
- 14. Reference Architecture - Connected Vehicles
- 15. Agenda 1. Industry Trends - Connected World, Industry 4.0 2. Manufacturing 3. Connected Vehicles 4. Mobility Services 5. Smart City 6. HiveMQ and Confluent
- 16. DB Musterfirma | Vorname Name | Abteilung | Datum ("Einfügen > Kopf- und Fußzeile") 22 Customer timetable Operational timetable Assignments Railway station knowledge Dispositions Train positions Matching Aggregation Consolidation Apache Kafka Analysis Railway station Trains Mobile Apps Employees Deutsche Bahn AG | Reisendeninformation RI-Plattform Single Point of Truth
- 17. ZF - Enabling Next Generation of Mobility Cloud Ecosystem for Connected Vehicles • Cloud-agnostic services without vendor lock-in HiveMQ Solution ● Open extension system for custom extensions ● Flexible deployment model to deploy Connectivity Suite to a public cloud or on-premise ● Implement authorization and authentication the way they wanted Results ● Remote diagnostics for technical insight & management of vehicle performance ● Fleet monitoring ● Secure & reliable middleware between connected vehicles & cloud services bit.ly/hivemq-zf
- 18. Reference Architecture - Mobility Services
- 19. Other Components Kafka Ecosystem (3) Read Data TensorFlow I/O (5) Deploy Model (2) Preprocess Data (8a) Alert Car Real Time Kafka App TensorFlow Serving gRPC Car Sensor HiveMQ MQTT Broker MQTT Connector Kafka Connect or Confluent Proxy or HiveMQ Plugin Kafka Cluster Kafka Connect Grafana Elastic Search KSQL TensorFlow (04) Train Model TensorFlow Real Time Kafka Streams Application (Java / Scala) Real Time Edge Computing (C / librdkafka) TensorFlow Lite (1) Ingest Data (8b) Alert Driver (e.g. mobile App) (6a) Consume Car Data (6b) All Data (7) Potential Defect 25
- 20. https://github.com/kaiwaehner/hivemq-mqtt-tensorflow-kafka-realtime-iot-machine-learning-training-inference or http://bit.ly/kafka-mqtt-ml-demo Try it out in 30 minutes! Demo 100.000 Connected Cars (Kafka + MQTT + TensorFlow) 26
- 21. Agenda 1. Industry Trends - Connected World, Industry 4.0 2. Manufacturing 3. Connected Vehicles 4. Mobility Services 5. Smart City 6. HiveMQ and Confluent
- 22. NAV (Norwegian Work and Welfare Department): Life is a Stream of Events https://www.confluent.io/kafka-summit-sf18/life-is-a-stream-of-events/
- 23. Berlex - Cloud-based Traffic Control Systems Portable, cloud-based traffic signals • Reliable control and monitoring of traffic signals from the cloud HiveMQ Solution ● Ability to run without TLS ● Support for MQTT 5 shared subscriptions ● Clusters can be configured with ease to achieve the high availability ● Consolidate log files of all cluster nodes onto a central Syslog server Results ● Successful deployment in Sweden and Norway planned European expansion ● Real-time information delivery & constant monitoring ● High level of customer satisfaction bit.ly/hivemq-berlex
- 24. Reference Architecture - Smart City
- 25. Agenda 1. Industry Trends - Connected World, Industry 4.0 2. Manufacturing 3. Connected Vehicles 4. Mobility Services 5. Smart City 6. HiveMQ and Confluent
- 26. Confluent Platform Fully Managed Cloud Service Self Managed Software FREEDOM OF CHOICE COMMITTER-DRIVEN EXPERTISE Partners Training Professional Services Enterpris e Support Apache Kafka EFFICIENT OPERATIONS AT SCALE PRODUCTION- STAGE PREREQUISITES UNRESTRICTED DEVELOPER PRODUCTIVITY SQL-based Stream Processing KSQL (ksqlDB) Rich Pre-built Ecosystem Connectors | Hub | Schema Registry Multi-language Development non-Java clients | REST Proxy GUI-driven Mgmt & Monitoring Control Center Flexible DevOps Automation Operator | Ansible Dynamic Performance & Elasticity Auto Data Balancer | Tiered Storage Enterprise-grade Security RBAC | Secrets | Audit logs Data Compatibility Schema Registry | Schema Validation Global Resilience Multi-Region Clusters | Replicator Developer Operator Architect Open Source | Community licensed PARTNERSHIP FOR BUSINESS SUCCESS Complete Engagement Model Revenue / Cost / Risk Impact TCO / ROI Executive Buyer
- 27. HiveMQ Platform
- 28. Kai Waehner Technology Evangelist [email protected] linkedin.com/in/kaiwaehner @KaiWaehner www.confluent.io www.kai-waehner.de Dominik Obermaier CTO HiveMQ [email protected] www.linkedin.com/in/dobermai @dobermai www.hivemq.com 34 Questions? Feedback? Please contact us!