The Rise Of Event Streaming – Why Apache Kafka Changes Everything
3 likes1,598 views
The rise of event streaming, particularly through Apache Kafka, revolutionizes how companies manage real-time data processing, shifting from traditional data storage to a more dynamic, event-driven architecture. This transition enables organizations to innovate and deliver enhanced customer experiences while improving operational efficiencies without the need for complete system overhauls. As every company increasingly becomes a software-centric entity, the adoption of event streaming technologies is critical for competitive success in the digital landscape.
![53
Improve
Customer
Experience
(CX)
Increase
Revenue
(make money)
Business
Value
Decrease
Costs
(save
money)
Core Business
Platform
Increase
Operational
Efficiency
Migrate to
Cloud
Mitigate Risk
(protect money)
Key Drivers
Strategic Objectives
(sample)
Fraud
Detection
IoT sensor
ingestion
Digital
replatforming/
Mainframe Offload
Connected Car: Navigation & improved
in-car experience: Audi
Customer 360
Simplifying Omni-channel Retail at
Scale: Target
Faster transactional
processing / analysis
incl. Machine Learning / AI
Mainframe Offload: RBC
Microservices
Architecture
Online Fraud Detection
Online Security
(syslog, log aggregation,
Splunk replacement)
Middleware
replacement
Regulatory
Digital
Transformation
Application Modernization: Multiple
Examples
Website / Core
Operations
(Central Nervous System)
The [Silicon Valley] Digital Natives;
LinkedIn, Netflix, Uber, Yelp...
Predictive Maintenance: Audi
Streaming Platform in a regulated
environment (e.g. Electronic Medical
Records): Celmatix
Real-time app
updates
Real Time Streaming Platform for
Communications and Beyond: Capital One
Developer Velocity - Building Stateful
Financial Applications with Kafka
Streams: Funding Circle
Detect Fraud & Prevent Fraud in Real
Time: PayPal
Kafka as a Service - A Tale of Security
and Multi-Tenancy: Apple
Example Use Cases
$↑
$↓
$
Example Case Studies
(of many)](https://image.slidesharecdn.com/oop2020keynoteeventstreamingkaiwaehner-200206165717/85/The-Rise-Of-Event-Streaming-Why-Apache-Kafka-Changes-Everything-47-320.jpg)
The Rise Of Event Streaming – Why Apache Kafka Changes Everything
- 1. 1 The Rise Of Event Streaming – Why Apache Kafka Changes Everything Kai Waehner | Technology Evangelist, Confluent [email protected] | LinkedIn | @KaiWaehner | www.confluent.io | www.kai-waehner.de
- 2. 22 The world is changing.
- 5. 5 Banking
- 6. 6 Largest media co - creates no content Largest movie house - owns no cinemas Online bookstore - diversifying across industries Largest taxi company - owns no vehicles Most valuable retailer - has no inventory World’s largest accommodation provider - owns no real estate
- 7. 77 The New Enterprise Reality Innovate or be disrupted ● Create new business models ● Deliver new, real-time customer experiences ● Deliver massive internal efficiencies Every company is a software company ● Capital One 10,000 of 40,000 employees are software engineers ● Goldman Sachs 1.5B (billion!) lines of code across 7,000+ applications ● JP Morgan: 50+% of IT budget for new projects and applications, <50% for maintenance Innovation is about events, not “data” ● Digital-first organizations must pick the right foundational technologies ● Established enterprises must implement new technologies without “rip-and-replace”
- 8. 8 Every Company is Becoming a Software Company
- 9. 9 Every Company is Becoming a Software Company
- 11. 11 The World has Changed Business Digitalization Trends are Driving the Need to Process Events at a whole new Scale, Speed and Efficiency
- 13. 1313 A company is built on DATA FLOWS but all we have is DATA STORES
- 14. 14 A company built on data stores looks like this
- 15. 15 ETL/Data Integration Messaging Highly Scalable Durable Persistent Ordered Real-time Difficult to Scale No Persistence After Consumption No Replay Batch Expensive Time Consuming
- 16. 1616 How is it done with an Event Streaming Platform
- 17. 1717 Highly Scalable Persistent ETL/Data Integration MessagingETL/Data Integration MessagingMessaging Batch Expensive Time Consuming Difficult to Scale No Persistence After Consumption No Replay Real-timeHighly Scalable Durable Persistent Ordered Real-time Event Streaming
- 18. 18 Events A Sale An Invoice A Trade A Customer Experience
- 19. 19 An Event Streaming Platform is the Underpinning of an Event-driven Architecture Microservices DBs SaaS apps Mobile Customer 360 Real-time fraud detection Data warehouse Producers Consumers Database change Microservices events SaaS data Customer experience s Streams of real time events Streaming AppsStreaming Apps Streaming Apps
- 20. 20
- 21. 21 The beginning of a new Era https://engineering.linkedin.com/distributed-systems/log-what-every-software-engineer-should-know-about-real-time-datas-unifying The first use case: Log Analytics. This is why Kafka was created!
- 22. 22 ● Global Scale ● Real-time ● Persistent Storage ● Stream Processing Apache Kafka: The De-facto Standard for Real-Time Event Streaming Edge Cloud Data LakeDatabases Datacenter IoT SaaS AppsMobile Microservices Machine Learning Apache Kafka
- 23. 23 Apache Kafka at Scale at Tech Giants > 7 trillion messages / day > 6 Petabytes / day “...you name it” * Kafka Is not just used by tech giants ** Kafka is not just used for big data
- 25. 25 Consuming from Kafka P Time C2 Near Real Time C3 Real Time C1 Batch
- 26. 26 Distributed System with Replication and High Availability Broker 1 Topic1 partition1 Broker 2 Broker 3 Broker 4 Topic1 partition1 Topic1 partition1 Leader Follower Topic1 partition2 Topic1 partition2 Topic1 partition2 Topic1 partition3 Topic1 partition4 Topic1 partition3 Topic1 partition3 Topic1 partition4 Topic1 partition4
- 27. 27 Distributed System with Replication and High Availability read, write App (Kafka Streams) Kafka (data) More Apps (KSQL, Connect, Python, REST, “You-name-it”) BookingsTeam FraudTeam … MobileTeam …
- 28. 28 Apache Kafka (kafka.apache.org) includes Kafka Connect and Kafka Streams Kafka Streams Your app sinksource KafkaConnect KafkaConnect
- 29. 2929 KSQL Continuous queries on infinite streams
- 30. 3030 STREAM PROCESSING Create and store materialized views Filter Analyze in-flight Time C CC
- 31. 31 TRADITIONAL DATABASE EVENT STREAM PROCESSING SELECT * FROM DB_TABLE CREATE TABLE T AS SELECT * FROM EVENT_STREAM Active Query: Passive Data: DB Table Active Data: Passive Query: Event Stream
- 33. 33 PUSH PULL APP Jay’s credit score is 670 Jay’s credit score is 710 Jay’s credit score is 695 What is Jay’s credit score now? 695 APP
- 34. 37 The stream is materialized into a table, which is emitted as a stream Payments Stream Credit Score Stream CREATE TABLE credit_scores AS SELECT user, updateScore(p.amount)…
- 35. 3838 What is the result
- 36. 4040 Event-Driven App (Location Tracking) Only Real-Time Events Messaging Queues can do this Contextual Event-Driven App (ETA) Real-Time combined with stored data Only Event Streaming Platforms can do this Where is my driver? When will my driver get here? Where is my driver? When will my driver get here? Why add the word Contextual?
- 37. 41 Front, rear and top view cameras Parking assistant Environment pointer Ultrasonic Sensors Parking assistant with front and rear camera plus environment indicator Crash Sensors Front protection adaptivity Side protection Tail impact protection Infrared Camera Rearview assistance with Pedestrian recognition Front and Rear Radar Sensors ACC with stop and go function Side assist Front Camera Audi Active lane assistant Speed limit indicator Adaptive light The Future of the Automotive Industry is a Real Time Data Cluster
- 38. 42 The Future of the Automotive Industry is a Real Time Data Cluster Front, rear and top view cameras Ultrasonic SensorsCrash Sensors Front Camera Infrared Camera Front and Rear Radar Sensors Traffic Alerts Hazard Alerts Personalization Anomaly Detection MQTT MQTT MQTT MQTT MQTTMQTT
- 39. 43
- 40. 44
- 41. 454545
- 42. 464646
- 43. 474747
- 44. 484848
- 45. 494949
- 46. 52 Real Time Machine Learning with Kafka and TensorFlow
- 47. 53 Improve Customer Experience (CX) Increase Revenue (make money) Business Value Decrease Costs (save money) Core Business Platform Increase Operational Efficiency Migrate to Cloud Mitigate Risk (protect money) Key Drivers Strategic Objectives (sample) Fraud Detection IoT sensor ingestion Digital replatforming/ Mainframe Offload Connected Car: Navigation & improved in-car experience: Audi Customer 360 Simplifying Omni-channel Retail at Scale: Target Faster transactional processing / analysis incl. Machine Learning / AI Mainframe Offload: RBC Microservices Architecture Online Fraud Detection Online Security (syslog, log aggregation, Splunk replacement) Middleware replacement Regulatory Digital Transformation Application Modernization: Multiple Examples Website / Core Operations (Central Nervous System) The [Silicon Valley] Digital Natives; LinkedIn, Netflix, Uber, Yelp... Predictive Maintenance: Audi Streaming Platform in a regulated environment (e.g. Electronic Medical Records): Celmatix Real-time app updates Real Time Streaming Platform for Communications and Beyond: Capital One Developer Velocity - Building Stateful Financial Applications with Kafka Streams: Funding Circle Detect Fraud & Prevent Fraud in Real Time: PayPal Kafka as a Service - A Tale of Security and Multi-Tenancy: Apple Example Use Cases $↑ $↓ $ Example Case Studies (of many)
- 48. 5454 How does Confluent help
- 49. $$€€££ - Making Money - Saving Money - Protecting Money Every Company is Becoming a Software Company
- 50. 5656 I N V E S T M E N T & T I M E VALUE 3 4 5 1 2 Event Streaming Maturity Model 56 Initial Awareness / Pilot Start to Build Pipeline / Deliver 1 New Outcome Leverage Stream Processing Build Contextual Event-Driven Apps Central Nervous System Product, Support, Training, Partners, Technical Account Management...
- 51. 5757 CONFLUENT PLATFORM EFFICIENT OPERATIONS AT SCALE PRODUCTION-STAGE PREREQUISITES UNRESTRICTED DEVELOPER PRODUCTIVITY Multi-language Development Rich Pre-built Ecosystem SQL-based Stream Processing GUI-driven Mgmt & Monitoring Flexible DevOps Automation Dynamic Performance & Elasticity Enterprise-grade Security Data Compatibility Global Availability APACHE KAFKA Fully Managed Cloud ServiceSelf Managed Software FREEDOM OF CHOICE COMMITTER-LED EXPERTISE PartnersTraining Professional Services Enterprise Support DEVELOPER OPERATOR ARCHITECT Hybrid Infrastructure
- 52. 58 Questions? Let’s connect... Kai Waehner Technology Evangelist [email protected] @KaiWaehner www.confluent.io www.kai-waehner.de LinkedIn