Aljoscha Krettek - Apache Flink® and IoT: How Stateful Event-Time Processing Enables Accurate Analytics
2 likes1,246 views
This document discusses Apache Flink and how it enables accurate analytics for Internet of Things (IoT) applications through stateful event-time stream processing. It begins by defining IoT and event-time stream processing, explaining that IoT data is continuously generated and has timestamps. It then discusses challenges like time mismatches between event time and processing time. The document also covers Flink's capabilities for stateful stream processing including failure handling through checkpoints, updating applications using savepoints, and high availability of the JobManager. It positions Flink as a stateful stream processor well-suited for IoT use cases.
More Related Content
Similar to Aljoscha Krettek - Apache Flink® and IoT: How Stateful Event-Time Processing Enables Accurate Analytics(20)
Aljoscha Krettek - Apache Flink® and IoT: How Stateful Event-Time Processing Enables Accurate Analytics
- 1. 1 Aljoscha Krettek @aljoscha ApacheCon NorthAmerica May, 2017 Apache Flink® and IoT: How Stateful Event-Time Processing Enables Accurate Analytics
- 2. What I’d Liketo Talk About 2 § IoT and event-time stream processing § Stateful stream processing § Streaming architecture and Flink
- 3. 3 Original creators of ApacheFlink® Providers of the dA Platform, a supported Flink distribution
- 4. IoT and Event-timeStream Processing 4
- 5.
- 6. A Simple Definition 6 IoTuse cases from the system’s perspective: A large number of (distributed) things continuously generating a large amount of data.
- 7. IoT: Some Insights 7 §Data is continuously produced → Stream Processing § Events have a timestamp → Event-time based processing § Data/Events can arrive with huge delays/out-of-order § Most analyses happen on time windows
- 8. What Is Event-TimeProcessing 8 1977 1980 1983 1999 2002 2005 2015 Processing Time Episode IV Episode V Episode VI Episode I Episode II Episode III Episode VII Event Time
- 9. What Is Event-TimeProcessing 9 1312735961112 1234567891011121314 Processing Time Event timestamp Message Queue
- 10. What’s The Problem? 10 13 12 735961112 1234567891011121314 ProcessingTime Processing-Time Windows 137356 12 137 356Event-Time Windows 12 1112 Mismatch between event time and processing time.
- 11. Sources of TimeMismatch § Big Mismatch • Network disconnects • Slow network § Small Mismatch • The nature of distributed systems • Differing system clock time 11
- 12. Small Event-Time Mismatch 12 RobustStream Processing with Apache Flink®: A Simple Walkthrough http://data-artisans.com/robust-stream-processing-flink-walkthrough/
- 13.
- 14.
- 15.
- 16. Recap: Event-Time § IoTuse cases need event-time processing § Even small mismatch of event time/processing time will lead to wrong results 16
- 17.
- 18. Stream Processing 18 Computation Computations on never-ending “streams”of data records (“events”)
- 19. Distributed Stream Processing 19 Computation Computation spreadacross many machines Computation Computation
- 20. Stateful Stream Processing 20 Computation State Stateis usually partitioned by some key in the data
- 21. Stateful Stream ProcessingII 21 § Result depends on history of stream § A stateful stream processor should gives the tools to manage state • Recover, roll back, version upgrade, etc.
- 22. 22 app state app state appstate event log Query service
- 23. Recap: Stateful Streams §Continuous processing of data that is continuously generated § I.e., pretty much all “big” data § It’s all about state and time § Flink does all of that 23
- 24.
- 25. Operational Questions § Whathappens in case of failures? § What if I need to update my code/Flink? § Can I re-process my data? § How can I execute my programs? 25
- 26. Failure Handling § JobManagerHigh-Availability using ZooKeeper § Periodic checkpoints of state to persistent storage (HDFS, S3, …) § In case of failure: rollback to previous consistent state 26
- 27. Savepoints § A persistentsnapshot of all state § When starting an application, state can be initialized from a savepoint § In-between savepoint and restore we can update Flink version or user code 27
- 28.
- 29. TL;DR § Stateful streamprocessing is nice 😎 § IoT use cases require proper time management § Apache Flink is a stateful stream processor with plenty of nifty features 29
- 30.
- 31.
- 32.
- 33. What Is Event-TimeProcessing 33 1977 1980 1983 1999 2002 2005 2015 Processing Time Episode IV Episode V Episode VI Episode I Episode II Episode III Episode VII Event Time
- 34. What Is Event-TimeProcessing 34 1312735961112 1234567891011121314 Processing Time Event timestamp Message Queue
- 35. What is Event-TimeStreaming § Events have timestamps § Processing depends on timestamps § An event-time stream processor should give you the tools to reason about time • Handle streams that are out of order 35 Your code state t3 t1 t2t4 t1-t2 t3-t4
- 36. Recap: Event-Time § IoTuse cases need event-time processing § Even small mismatch of event time/processing time will lead to wrong results 36
- 37.
- 38. A brief Historyof Flink 38 January ‘10 December ‘14 v0.5 v0.6 v0.7 March ‘16 Flink Project Incubation Top Level Project v0.8 v0.10 Release 1.0 Project Stratosphere (Flink precursor) v0.9 April ‘14
- 39. A brief Historyof Flink 39 January ‘10 December ‘14 v0.5 v0.6 v0.7 March ‘16 Flink Project Incubation Top Level Project v0.8 v0.10 Release 1.0 Project Stratosphere (Flink precursor) v0.9 April ‘14 The academia gap: Reading/writing papers, teaching, worrying about thesis Realizing this might be interesting to people beyond academia (even more so, actually)