Resilient Real-time Data Streaming across the Edge and Hybrid Cloud with Apache Kafka
1 like841 views
The document discusses resilient real-time data streaming solutions using Apache Kafka in edge and hybrid cloud environments, with various use cases, including automotive IoT, banking, and retail customer experiences. It highlights the importance of disaster recovery, multi-region infrastructures, and real-time analytics to improve operational efficiency and enhance customer engagement. The agenda covers various aspects of resilient enterprise architectures and provides examples of implementation in several industries.
Resilient Real-time Data Streaming across the Edge and Hybrid Cloud with Apache Kafka
- 1. Resilient Real-time Data Streaming across the Edge and Hybrid Cloud Use Cases, Architectures, and Examples for Data in Motion powered by Apache Kafka Kai Waehner Field CTO [email protected] linkedin.com/in/kaiwaehner confluent.io kai-waehner.de @KaiWaehner
- 2. kai-waehner.de @KaiWaehner – Resilient Data Streaming at Edge and Hybrid Cloud Agenda 1) Resilient enterprise architectures 2) Real-time data streaming with the Apache Kafka ecosystem 3) Cloud-first and serverless Industrial IoT in automotive 4) Multi-region infrastructure for core banking 5) Hybrid cloud for customer experiences in retail 6) Disconnected edge for safety and security in the public sector
- 3. kai-waehner.de @KaiWaehner – Resilient Data Streaming at Edge and Hybrid Cloud Agenda 1) Resilient enterprise architectures 2) Real-time data streaming with the Apache Kafka ecosystem 3) Cloud-first and serverless Industrial IoT in automotive 4) Multi-region infrastructure for core banking 5) Hybrid cloud for customer experiences in retail 6) Disconnected edge for safety and security in the public sector
- 4. kai-waehner.de @KaiWaehner – Resilient Data Streaming at Edge and Hybrid Cloud AWS Cloud Outage hit Disney World Visitors… https://www.cnet.com/tech/services-and-software/disney-parks-were-already-facing-heat-from-fans-then-an-aws-outage-came-along/
- 5. kai-waehner.de @KaiWaehner – Resilient Data Streaming at Edge and Hybrid Cloud Why one data center or cloud region is not good enough? Latency / Cost Disaster Recovery Privacy / Compliance
- 6. kai-waehner.de @KaiWaehner – Resilient Data Streaming at Edge and Hybrid Cloud Disaster Recovery – RPO and RTO RPO = Recovery Point Objective RTO = Recovery Time Objective
- 7. kai-waehner.de @KaiWaehner – Resilient Data Streaming at Edge and Hybrid Cloud ZERO RPO requires synchronous replication ZERO RTO requires seamless failover
- 8. kai-waehner.de @KaiWaehner – Resilient Data Streaming at Edge and Hybrid Cloud Agenda 1) Resilient enterprise architectures 2) Real-time data streaming with the Apache Kafka ecosystem 3) Cloud-first and serverless Industrial IoT in automotive 4) Multi-region infrastructure for core banking 5) Hybrid cloud for customer experiences in retail 6) Disconnected edge for safety and security in the public sector
- 9. kai-waehner.de @KaiWaehner – Resilient Data Streaming at Edge and Hybrid Cloud Real-time Data in Motion beats Slow Data. Transportation Predictive maintenance Driver-rider match ETA updates Banking Instant payments Fraud detection Mobile applications / customer experience Retail Real-time inventory Real-time POS reporting Personalization Entertainment Real-time recommendations Personalized news feed In-car purchases
- 10. kai-waehner.de @KaiWaehner – Resilient Data Streaming at Edge and Hybrid Cloud Apache Kafka is the Platform for Data in Motion MES ERP Sensors Mobile Customer 360 Real-time Alerting System Data warehouse Producers Consumers Streams and storage of real time events Stream processing apps Connectors Connectors Stream processing apps Supplier Alert Forecast Inventory Customer Order 11
- 11. kai-waehner.de @KaiWaehner – Resilient Data Streaming at Edge and Hybrid Cloud Apache Kafka = A Resilient, Distributed System Broker 1 Topic1 partition1 Broker 2 Broker 3 Broker 4 Topic1 partition1 Topic1 partition1 Leader Follower Topic1 partition2 Topic1 partition2 Topic1 partition2 Topic1 partition3 Topic1 partition4 Topic1 partition3 Topic1 partition3 Topic1 partition4 Topic1 partition4
- 12. kai-waehner.de @KaiWaehner – Resilient Data Streaming at Edge and Hybrid Cloud Resilient Data Streaming across Edge and Hybrid Cloud Streaming Replication between Kafka Clusters Bridge to Databases, Data Lakes, Apps, APIs, SaaS Aggregate Small Footprint Edge Deployments with Replication (Aggregation) Simplify Disaster Recovery Operations with Multi-Region Clusters for RPO=0 and RTO~0 Stream Data Globally with Replication and Cluster Linking 16
- 13. kai-waehner.de @KaiWaehner – Resilient Data Streaming at Edge and Hybrid Cloud Shipping Industry Marine, Oil Transport, Vessel Fleet, Shipping Line, Drones Real-time Operations, Logistics, Predictive Maintenance, Security Customer Data Crew, Cargo Vessel Data Fuel Consumption, Speed, Planned Maintenance Automatic Identification System (AIS) Unique Identification, Position, Course, Weather, Draft Drone Data Deliveries, Survey/Inspection of Assets such as Oil Rigs, Pipelines, Offshore Turbines Edge Analytics Bidirectional Edge to Cloud Integration Data Ingestion Stream Processing Data Integration Logistics Track&Trace Routing Monitoring Alerting Command&Control Batch Analytics Reporting Machine Learning Backend Systems Oracle, SAP, OSIsoft PI, etc. X = Event Streaming X = Other Technologies Bi-Directional Hybrid Cloud Replication
- 14. kai-waehner.de @KaiWaehner – Resilient Data Streaming at Edge and Hybrid Cloud Agenda 1) Resilient enterprise architectures 2) Real-time data streaming with the Apache Kafka ecosystem 3) Cloud-first and serverless Industrial IoT in automotive 4) Multi-region infrastructure for core banking 5) Hybrid cloud for customer experiences in retail 6) Disconnected edge for safety and security in the public sector
- 15. kai-waehner.de @KaiWaehner – Resilient Data Streaming at Edge and Hybrid Cloud BMW Group Mission-critical workloads across the edge and cloud • Why Kafka? Decoupling. Transparency. Innovation. • Why Confluent? Stability is key in manufacturing • Decoupling between logistics and production systems • Cloud-first event streaming on Azure Cloud with serverless Confluent Cloud • Use case • Logistics and supply chain in global plants • Right stock in place (physically and in ERP systems like SAP) • Just in time, just in sequence • Lot of critical applications 19 Jay Kreps, Confluent CEO Felix Böhm, BMW Plant Digitalization and Cloud Transformation Keynote at Kafka Summit Eurpoe 2021: https://www.youtube.com/watch?v=3cG2ud7TRs4
- 16. kai-waehner.de @KaiWaehner – Resilient Data Streaming at Edge and Hybrid Cloud Condition Monitoring and Predictive Maintenance Stateless and stateful stream processing for real-time data correlation with Kafka-native tools (Kafka Streams / ksqlDB) 8 9 1 0 1 1 1 2 1 3 1 4 1 5 1 6 3 4 5 6 7 2 1 Time Sensor Events
- 17. kai-waehner.de @KaiWaehner – Resilient Data Streaming at Edge and Hybrid Cloud Condition Monitoring and Predictive Maintenance Stateless and stateful stream processing for real-time data correlation with Kafka-native tools (Kafka Streams / ksqlDB) 8 9 1 0 1 1 1 2 1 3 1 4 1 5 3 4 5 6 7 2 1 Time Sensor Events 1 6 Condition Monitoring (Temperature Spikes) Stateless Filter Above-Threshold Events Streams builder .stream(”temperature-sensor") .filter((key, sensor-data) -> sensor-data.temperature > 100) .to(”temperature-spikes");
- 18. kai-waehner.de @KaiWaehner – Resilient Data Streaming at Edge and Hybrid Cloud Condition Monitoring and Predictive Maintenance Stateless and stateful stream processing for real-time data correlation with Kafka-native tools (Kafka Streams / ksqlDB) 8 9 1 0 1 1 1 2 1 3 1 4 1 5 1 6 3 4 5 6 7 2 1 Time Sensor Events Predictive Maintenance (Continuous Anomaly Detection) Stateful Correlation of Events CREATE TABLE anomaly_detection AS SELECT temperature_spike_id, COUNT(*) AS total_spikes, AVG(temperature) AS avg_temperature FROM sensor-data WINDOW TUMBLING (SIZE 1 HOUR) GROUP BY temperature_spike_id EMIT CHANGES;
- 19. kai-waehner.de @KaiWaehner – Resilient Data Streaming at Edge and Hybrid Cloud Condition Monitoring and Predictive Maintenance Stateless and stateful stream processing for real-time data correlation with Kafka-native tools (Kafka Streams / ksqlDB) 8 9 1 0 1 1 1 2 1 3 1 4 1 5 1 6 3 4 5 6 7 2 1 Time Sensor Events Predictive Maintenance (Continuous Anomaly Detection) Real-time Machine Learning CREATE STREAM anomaly_detection AS SELECT sensor_id, detect_anomaly(sensor_values) FROM machine; TensorFlow model embedded in User Defined Function (UDF)
- 20. kai-waehner.de @KaiWaehner – Resilient Data Streaming at Edge and Hybrid Cloud Agenda 1) Resilient enterprise architectures 2) Real-time data streaming with the Apache Kafka ecosystem 3) Cloud-first and serverless Industrial IoT in automotive 4) Multi-region infrastructure for core banking 5) Hybrid cloud for customer experiences in retail 6) Disconnected edge for safety and security in the public sector
- 21. kai-waehner.de @KaiWaehner – Resilient Data Streaming at Edge and Hybrid Cloud Disaster Recovery @ JPMorgan https://www.confluent.io/kafka-summit-san-francisco-2019/secure-kafka-at-scale-in-true-multi-tenant-environment
- 22. kai-waehner.de @KaiWaehner – Resilient Data Streaming at Edge and Hybrid Cloud Multi-Region Kafka Cluster in Financial Services Zero downtime + zero data loss (RPO=0 and RTO~0) + automated disaster recovery Large Bank Transaction Log Transaction Log Location Location synchronous asynchronous ● ‘Payment’ transactions enter from us-east and us-west with fully synchronous replication ● ‘Log’ and ‘Location’ information in the same cluster use async - optimized for latency ● Automated disaster recovery (zero downtime, zero data loss) Result: Clearing time from ‘deposit’ to ‘available’ goes from 5 days to 5 seconds (including security checks) (Only available in Confluent Platform) Hundreds of miles distance between the data centers
- 23. kai-waehner.de @KaiWaehner – Resilient Data Streaming at Edge and Hybrid Cloud Migration with Cluster Linking 27
- 24. kai-waehner.de @KaiWaehner – Resilient Data Streaming at Edge and Hybrid Cloud Robinhood Mission: “Democratize finance for all” Kafka for mission-critical and analytics use cases Microservices using various technologies 28 https://www.confluent.io/events/kafka-summit-americas-2021/taming-a-massive-fleet-of-python-based-kafka-apps-at-robinhood/
- 25. kai-waehner.de @KaiWaehner – Resilient Data Streaming at Edge and Hybrid Cloud Thought Machine – Core Banking • Cloud-native core banking software • Transactional workloads (24/7, zero data loss) • Flexible product engine powered by smart contracts (not blockchain) 29 https://www.confluent.io/events/kafka-summit-apac-2021/scaling-a-core-banking-engine-using-apache-kafka/
- 26. kai-waehner.de @KaiWaehner – Resilient Data Streaming at Edge and Hybrid Cloud “Transactions” in Apache Kafka 30 Exactly-Once Semantics (EOS) available since Kafka 0.11 (June 2017): https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Once+Delivery+and+Transactional+Messaging https://www.confluent.io/kafka-summit-london18/dont-repeat-yourself-introducing-exactly-once-semantics-in-apache-kafka/
- 27. kai-waehner.de @KaiWaehner – Resilient Data Streaming at Edge and Hybrid Cloud Agenda 1) Resilient enterprise architectures 2) Real-time data streaming with the Apache Kafka ecosystem 3) Cloud-first and serverless Industrial IoT in automotive 4) Multi-region infrastructure for core banking 5) Hybrid cloud for customer experiences in retail 6) Disconnected edge for safety and security in the public sector
- 28. kai-waehner.de @KaiWaehner – Resilient Data Streaming at Edge and Hybrid Cloud Royal Caribbean - Offline Edge for Swimming Retail Stores https://www.confluent.io/kafka-summit-lon19/seamless-guest-experience-with-kafka-streams/
- 29. kai-waehner.de @KaiWaehner – Resilient Data Streaming at Edge and Hybrid Cloud CRM 3rd party payment provider Context-specific real-time upsell Customer data Payment processing and fraud detection as a service Manager Get report API Customer Customer Customer data Train schedule Payment data Loyalty information Streams of real time events Customer data Train schedule Payment data Loyalty information Streams of real time events Customer data Train schedule Payment data Loyalty information Streams of real time events Hybrid Retail Architecture
- 30. kai-waehner.de @KaiWaehner – Resilient Data Streaming at Edge and Hybrid Cloud Point of Sale (POS) Loyalty System Local Inventory Management Payment Discount Customer data Train schedule Payment data Loyalty information Streams of real time events Global Inventory Management Event Streaming at the Edge in the Smart Retail Store Item Availability
- 31. kai-waehner.de @KaiWaehner – Resilient Data Streaming at Edge and Hybrid Cloud Omnichannel Retail Time P C3 C2 C1 Sales Talk on site in Car Dealership Right now Location-based Customer Action Customer 360 (Website, Mobile App, On Site in Store, In-Car) Car Configurator 10 and 8 days ago Context-specific Marketing Campaign 90 and 60 days ago
- 32. kai-waehner.de @KaiWaehner – Resilient Data Streaming at Edge and Hybrid Cloud Agenda 1) Resilient enterprise architectures 2) Real-time data streaming with the Apache Kafka ecosystem 3) Cloud-first and serverless Industrial IoT in automotive 4) Multi-region infrastructure for core banking 5) Hybrid cloud for customer experiences in retail 6) Disconnected edge for safety and security in the public sector
- 33. kai-waehner.de @KaiWaehner – Resilient Data Streaming at Edge and Hybrid Cloud Data Processing at the Edge Time P C1 C2 C3 Know-your-customer Loyalty app, predictive behavior, … Estimated time of arrival Connect to the gaming server for kids Play games, earn rewards, communicate with other kids in the train, … Always on (even “offline”) Replayability Cost-efficiency Low latency
- 34. kai-waehner.de @KaiWaehner – Resilient Data Streaming at Edge and Hybrid Cloud Devon Energy Oil & Gas Industry Improve drilling and well completion operations Edge stream processing/analytics + closed-loop control ready Vendor agnostic (pumping, wireline, coil, offset wells, drilling operations, producing wells) Replication to the cloud in real-time at scale Cloud agnostic (AWS, GCP, Azure) Source: Energy in Data - Powered by AAPG, SEG & SPE: energyindata.org
- 35. kai-waehner.de @KaiWaehner – Resilient Data Streaming at Edge and Hybrid Cloud Smart Soldiers at the Edge 39 Sensor A Sensor B Sensor X MQTT Confluent Platform (Single Broker) Single Kafka broker deployed on a small computer and leveraging Cluster Linking to publish sensor data to the Command Post. Command Post running Confluent Platform aggregating information from soliders and other sensor data Weather Personnel Logistics Targets Sensor data published to Command Post when connected to network Enhanced Situational Awareness
- 36. kai-waehner.de @KaiWaehner – Resilient Data Streaming at Edge and Hybrid Cloud Why people choose Confluent for building resilient architectures?
- 37. kai-waehner.de @KaiWaehner – Resilient Data Streaming at Edge and Hybrid Cloud Car Engine Car Self-driving Car Confluent completes Apache Kafka. Cloud-native. Everywhere.