GNW03: Stream Processing with Apache Kafka by Gwen Shapira
5 likes3,153 views
The document discusses the significance of stream processing and how Apache Kafka serves as a key technology for managing real-time data streams. It highlights Kafka's role as a log storage system that allows for the continuous processing and availability of data, emphasizing its advantages in scalability, fault tolerance, and simplification of stream processing tasks. Additionally, it presents the concept of table-stream duality, illustrating the relationship between events and the current state of data.
GNW03: Stream Processing with Apache Kafka by Gwen Shapira
- 3. I’ll tell you about • What is stream processing and why it matters • What is Apache Kafka • How Kafka helps stream processing Stay awake for this part
- 6. Stream Processing Paradigm • Data is generated at its own rate as “Streams” • We can process as much or as little as we want • Continuously • Results are available in real-time • But nothing waits for specific results • Time for data availability? • More than “few ms” • Less than “hours”
- 8. This is the world changing bit • Most of the business is… • Not urgent enough to require immediate response • But can’t wait for the next day • “Streams of events” represents something fundamental • Same way relational tables are fundamental
- 22. Method 2: The Stream Processing Frameworks • Storm • Spark • Flink • Samza • Apex • Nifi • StreamBase • InfoSphere Streams • Google DataFlow (AKA Beam) • I can go on for 5 more pages…
- 23. Few of those are really popular! • Pro: They handle some hard problems • Con: It can be too complex
- 24. What do I mean by too complex? Hadoop Cluster II Storage Processing SolR Hadoop Cluster I ClientClient Flume Agents Hbase / Memory Spark Streaming HDFS Hive/Imp ala Map/Red uce Spark Search Automated & Manual Analytical Adjustments and Pattern detection Fetching & Updating Profiles Adjusting NRT Stats HDFSEventSink SolR Sink Batch Time Adjustments Automated & Manual Review of NRT Changes and Counters Local Cache Kafka Clients: (Swipe here!) Web App
- 27. Enter KafkaStreams 3 Simplifications: 1. Uses Kafka 2. No Framework 3. Unify Tables and Streams
- 30. Handling Time
- 31. No Framework • It is just a library that does transformations • We can add languages on top • Kafka does everything we needed the framework to do • You don’t need “framework” to run queries, why do you need it to run queries continuously?
- 46. Where can we learn more? • http://www.confluent.io/blog • http://kafka.apache.org/docume ntation.html • http://docs.confluent.io/current