Data Driven Enterprise with Apache Kafka
3 likes657 views
The document discusses real-time streaming with Apache Kafka. It introduces Bill McLaren, a streaming data engineer at Confluent. It then provides background on Confluent, which was founded by the creators of Apache Kafka and has many Kafka committers. The rest of the document discusses challenges with traditional infrastructure-driven systems and how a data-driven approach using Apache Kafka's distributed event streaming platform can help by processing and storing streams of data in real-time.
Data Driven Enterprise with Apache Kafka
- 1. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Bill McLaren Streaming Data Engineer, Confluent Data Driven Enterprise Real-time Streaming with Apache Kafka®
- 2. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. 80% of active Kafka Committers Founded September 2014 Technology developed while at LinkedIn Founded by the creators of Apache Kafka About Confluent™ Jay Kreps CEO Cheryl Dalrymple CFO Todd Barnett VP WW Sales Neha Narkhede CTO, VP Engineering
- 3. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Infrastructure Driven Enterprise
- 4. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Challenges In Infrastructure Driven Enterprise Divergent • “Normalization” • Combined & Derived • Which one is the source of truth? • What happens if any attributes change? Duplicate • Applications • Orgs • Data centers Slow • Batch processes • Multiple hops add up • Slow in propagating changes Serial • How many hops? • How many different processes? • What happens if one breaks
- 5. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. What is the cost Agility Cost Success
- 6. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Data Driven Enterprise Collapse the space between the data and the mission Time | Distance
- 7. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Data Driven Enterprise Think Event Streaming
- 8. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Apache Kafka: A Distributed Event Streaming Platform
- 9. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Data Driven Enterprise You have probably seen a diagram like that before
- 10. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Apache Kafka: An Event Streaming Platform Process streams of data in real time, as they occur 110101 010111 001101 100010 Publish and subscribe to streams of data similar to a message queue or enterprise messaging system 110101 010111 001101 100010 Store streams of data in a distributed highly available fault tolerant way 110101 010111 001101 100010
- 11. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Open Open Source Multi Cloud & On Premise Explosive Ecosystem
- 12. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Confluent Platform: a Streaming Platform based on Apache Kafka
- 13. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Please complete the session survey in the summit mobile app.
- 14. © 2018, Amazon Web Services, Inc. or its affiliates. All rights reserved. Thank you!