Apache Flink for IoT: How Event-Time Processing Enables Easy and Accurate Analytics by Aljoscha Krettek
3 likes2,315 views
This document discusses Apache Flink for IoT event-time stream processing. It begins by introducing streaming architectures and Flink. It then discusses how IoT data has important properties like continuous data production and event timestamps that require event-time based processing. Examples are provided of companies like King and Bouygues Telecom using Flink for billions of events per day with challenges like out-of-order data and flexible windowing. Event-time processing in Flink is able to handle these challenges through features like watermarks.
Apache Flink for IoT: How Event-Time Processing Enables Easy and Accurate Analytics by Aljoscha Krettek
- 2. 1! Aljoscha Krettek @aljoscha Big Data Spain November 17, 2016 Apache Flink for IoT: How Event-Time Processing Enables Easy and Accurate Analytics
- 3. What I’d Like to Talk About 2 § Streaming Architecture and Flink § IoT and Event-Time based stream processing § Use-Case Examples
- 4. 3 Original creators of Apache Flink® Providers of the dA Platform, a supported Flink distribution
- 5. Intro: The Streaming Architecture 4
- 6. Rethinking Data Architecture § Better app isolation § Real-time reaction to events § Robust continuous applications § Process both real-time and historical data 5
- 7. 6 app state app state app state event log Query service
- 8. What is (Distributed) Streaming § Streaming: Computations on never- ending “streams” of data records (“events”) § Distributed: Computation spread across many machines 7 Your code Your code Your code Your code
- 9. What is Stateful Streaming § Computation and state • E.g., counters, windows of past events, state machines, trained ML models § Result depends on history of stream § A stateful stream processor should gives the tools to manage state • Recover, roll back, version, upgrade, etc 8 Your code state
- 10. What is Event-Time Streaming § Data records associated with timestamps (time series data) § Processing depends on timestamps § An event-time stream processor should give you the tools to reason about time • Handle streams that are out of order • Core feature is watermarks – a clock to measure event time 9 Your code state t3 t1 t2t4 t1-t2 t3-t4
- 11. Recap: What is Streaming? § Continuous processing on data that is continuously generated § I.e., pretty much all “big” data § It’s all about state and time § Flink does all of what we just saw 10
- 12. IoT and Event-time Stream Processing 11
- 13. 12 1read.bi/1yDOQQ3 The 'Internet Of Everything' Will Generate $14.4 Trillion Of Value Over The Next Decade.1
- 15. A Simple Definition 14 IoT use cases from the system’s perspective: A large number of (distributed) things generating a large amount of data.
- 16. Important Properties 15 § Data is continuously produced → Stream Processing § Events have a timestamp that has to be considered → Event-time based processing § Data/Events can arrive with huge delays § Most analyses happen on time windows
- 17. Remember: Streaming technology is enabling the obvious: continuous processing on data that is continuously produced Hint: you already have streaming data 16
- 18. What Is Event-Time Processing 17 1312735961112 1234567891011121314 Processing Time Event timestamp Message Queue
- 19. What’s The Problem? 18 13 12 735961112 1234567891011121314 Processing Time Processing-Time Windows 137356 12 137 356Event-Time Windows 12 1112 Mismatch between event time and processing time.
- 20. Sources of Time Mismatch § Big Mismatch • Network disconnects • Slow network § Small Mismatch • The nature of distributed systems • Differing system clock time 19
- 21. Big Event-Time Mismatch 20 1977 1980 1983 1999 2002 2005 2015 Processing Time Episode IV Episode V Episode VI Episode I Episode II Episode III Episode VII Event Time
- 22. Small Event-Time Mismatch 21 Robust Stream Processing with Apache Flink®: A Simple Walkthrough http://data-artisans.com/robust-stream-processing-flink-walkthrough/
- 23. 22
- 24. 23
- 25. 24
- 26. Recap: Event-Time § IoT use cases need event-time processing § Even small mismatch of event time/ processing time will lead to wrong results 25
- 28. 30 Flink applications in production for more than one year. 10 billion events (2TB) processed daily Complex jobs of > 30 operators running 24/7, processing 30 billion events daily, maintaining state of 100s of GB with exactly-once guarantees Largest job has > 20 operators, runs on > 5000 vCores in 1000-node cluster, processes millions of events per second 27
- 29. King § Challenges: • Many games (Candy Crush, Farm Heroes, Pet Rescue, and Bubble Witch…) • 300 million monthly unique users • 30 billion events received every day § Need Event-Time Based statistics 28https://techblog.king.com/rbea-scalable-real-time-analytics-king/
- 31. Solution: RBEA § Multiplexing of multiple data scientist requests into a single Flink job § Groovy as language for analysis scripts § Event-time windowing 30https://techblog.king.com/rbea-scalable-real-time-analytics-king/
- 32. Bouygues Telecom 31http://flink-forward.org/kb_sessions/a-brief-history-of-time-with-apache-flink-real-time-monitoring-and-analysis-with-flink-kafka-hb/ ~120 users* 5 Flink Production Apps 750 TB Storage 4 billion Events/ day 2015 ~300 users* 30 Flink Production Apps 2 PB Storage5 10 billion Events/ day 2016 * Users of the information system
- 33. Bouygues: Challenges § Low latency & streaming fashion counters § Massive amounts of data + bursty loads § Reliability § Multiple flow correlation § Time management: • Out of order & late events → our worst enemies • Flexible window management 32http://flink-forward.org/kb_sessions/a-brief-history-of-time-with-apache-flink-real-time-monitoring-and-analysis-with-flink-kafka-hb/
- 35. In Summary 34 § If you need to ask: you already have a streaming use case! § IoT requires Proper Time Management § Apache Flink has done that for a long time now* * Since version 0.10
- 37. 36 One day of hands-on Flink training One day of conference Tickets are on sale Call for Papers is already open Please visit our website: http://sf.flink-forward.org Follow us on Twitter: @FlinkForward
- 39. Appendix 38