The Past, Present, and Future of Apache Flink
Download as PPTX, PDF•0 likes•64 views
The document provides an overview of the evolution and capabilities of Apache Flink from its inception in 2014, focusing on its advancements in real-time and batch processing, state management, and high throughput. It discusses features such as event-time processing, user-defined functions, and the unification of batch and streaming analytics. Looking forward, the document highlights ongoing developments in Flink related to resource management, application scalability, and the integration of SQL capabilities for data-driven applications.
![© 2018 data Artisans16
More than just analytics: ProcessFunction
class MyFunction extends ProcessFunction[MyEvent, Result] {
// declare state to use in the program
lazy val state: ValueState[CountWithTimestamp] = getRuntimeContext().getState(…)
def processElement(event: MyEvent, ctx: Context, out: Collector[Result]): Unit = {
// work with event and state and schedule timers
}
def onTimer(timestamp: Long, ctx: OnTimerContext, out: Collector[Result]): Unit = {
// handle callback when event-/processing- time instant is reached
}
}
● ProcessFunction gives access to state, time
and events
● Low level API
● Enables data-driven applications](https://image.slidesharecdn.com/flinkforward2018-thepastpresentandfutureofapacheflink-181003122056/85/The-Past-Present-and-Future-of-Apache-Flink-16-320.jpg)
The Past, Present, and Future of Apache Flink
- 1. Aljoscha Krettek, engineering manager & co-founder Till Rohrmann, engineering manager & co-founder The Past, Present, and Future of Apache Flink®
- 2. © 2018 data Artisans2 Past
- 3. © 2018 data Artisans3 It all started in 2014 2009 - 2014 since 2014 ● Batch processor on top of streaming runtime ● First Apache Flink 0.6.0 release August 2014
- 4. © 2018 data Artisans4 August 2014 Batch processing
- 5. © 2018 data Artisans5 Flink learns to stream in real time DataStream API Stream Processing DataSet API Batch Processing Runtime Distributed Streaming Data Flow
- 6. © 2018 data Artisans6 ● Continuous & real-time November 2014 Batch processing Stream processing
- 7. © 2018 data Artisans7 Flink learns to remember
- 8. © 2018 data Artisans8 Flink learns to remember
- 9. © 2018 data Artisans9 Flink learns to remember
- 10. © 2018 data Artisans10 Flink learns to remember Remember where we left off
- 11. © 2018 data Artisans11 ● Continuous & real-time ● Stateful & exactly once June 2015 Batch processing Stream processing
- 12. © 2018 data Artisans12 Latency vs. Throughput? high latency low high throughput Prevailing belief ● 10s of millions of events/s ● Latency down to 1 ms ≠
- 13. © 2018 data Artisans13 Flink becomes event-time aware Episode IV Episode V Episode VI Episode I Episode II Episode III Episode VII Episode VIII Processing time 1977 1980 1983 1999 2002 2005 2015 2017 Flink becomes event-time aware
- 14. © 2018 data Artisans14 Flink becomes event-time aware Episode I Episode II Episode III Episode IV Episode V Episode VI Episode VII Episode VIII Processing time 1999 2002 2005 1977 1980 1983 2015 2017 Processing time Event time
- 15. © 2018 data Artisans15 ● Continuous & real-time ● Stateful & exactly once ● High throughput & low latency ● Event time November 2015 Batch processing Stream processing
- 16. © 2018 data Artisans16 More than just analytics: ProcessFunction class MyFunction extends ProcessFunction[MyEvent, Result] { // declare state to use in the program lazy val state: ValueState[CountWithTimestamp] = getRuntimeContext().getState(…) def processElement(event: MyEvent, ctx: Context, out: Collector[Result]): Unit = { // work with event and state and schedule timers } def onTimer(timestamp: Long, ctx: OnTimerContext, out: Collector[Result]): Unit = { // handle callback when event-/processing- time instant is reached } } ● ProcessFunction gives access to state, time and events ● Low level API ● Enables data-driven applications
- 17. © 2018 data Artisans17 ● Continuous & real-time ● Stateful & exactly once ● High throughput & low latency ● Event time February 2017 Batch processing Stream processing Data-driven applications
- 18. © 2018 data Artisans18 Present & Future
- 19. © 2018 data Artisans19 Hardening Faster network stack Application level flow control Resolving dependency hell Present in a nutshell Scaling Incremental snapshots Local recovery Scalable timers Interoperability Resource elasticity REST client-server interface Container entrypoint Stream SQL SQL client User-defined functions More powerful joins Misc State TTL Broadcast state Kafka exactly-once producer
- 20. © 2018 data Artisans20 Large, larger, Flink Time State Incremental snapshots ● Snapshot only state diff ● Incremental snapshots allow to handle very large state
- 21. © 2018 data Artisans21 Faster failover is always better
- 22. © 2018 data Artisans22 Varying workloads • Violating SLAs vs. wasting money • Varying workloads require to adapt resources
- 23. © 2018 data Artisans23 Revamped distributed architecture ResourceManager ClusterManager TaskManagerJobManager Dispatcher Client 1. Submit job 2. Start job 3. Request slots 4. Allocate resources 5. Start TaskManager 6. Execute job ● Support for full resource elasticity ● Application parallelism can be dynamically changed
- 24. © 2018 data Artisans24 ● Continuous & real-time ● Stateful & exactly once ● High throughput & low latency ● Event time ● Applications as first class citizens Present & Future Batch processing Stream processing Data-driven applications
- 25. © 2018 data Artisans25 Flink as a library (and still as a framework) • Deploying Flink applications should be as easy as starting a process • Bundle application code and Flink into a single image • Process connects to other application processes and figures out its role • Removing the cluster out of the equation P2 P3 P1 P4 New process
- 26. © 2018 data Artisans26 How much control do I need? Batch processing Continuous processing Real-time & data-driven applications ● Multiple short lived stages ● Different resource requirements per stage ● Efficient execution requires control over resources ● Flink allocates actively resources ● Continuously processing operators ● Constrained by external systems, SLAs and application logic ● External system can assign resources ● Flink reacts to available resources
- 27. © 2018 data Artisans27 Active vs. reactive mode • Active mode ‒ Flink is aware of underlying cluster framework ‒ Flink allocate resources ‒ E.g. existing YARN and Mesos integration • Reactive mode ‒ Flink is oblivious to its runtime environment ‒ External system allocates and releases resources ‒ Flink scales with respect to available resources ‒ Relevant for environments: Kubernetes, Docker, as a library
- 28. © 2018 data Artisans28 Scaling automatically • Latency • Throughput • Resource utilization • Connector signals
- 29. © 2018 data Artisans29 How we create Flink Jobs Flink APIs Stream/Batch Processing Runtime Distributed Streaming Data Flow Java/Scala
- 30. © 2018 data Artisans30 Flink SQL Flink APIs Stream/Batch Processing Runtime Distributed Streaming Data Flow Java/Scala SQL “NO CODING REQUIRED” Source/Sink definition in YAML Configuration in YAML SQL commandline User-defined functions Streaming and Batch Event time and processing time *since Flink 0.9.0 (June 2015)
- 31. © 2018 data Artisans31
- 32. © 2018 data Artisans32 “Join” me for some trading buy buy sell buy Join $ 17 £ 42 12.5 buy sell ₪
- 33. © 2018 data Artisans33 Introducing Time-versioned Table Joins buy buy sell buy Join buy sell curr rate time £ 42 3 £ 12 17 1453 31753 14 event time
- 34. © 2018 data Artisans34 SQL for pattern analysis? SELECT * from ?
- 35. © 2018 data Artisans35 Introducing MATCH_RECOGNIZE SELECT * FROM TaxiRides MATCH_RECOGNIZE ( PARTITION BY driverId ORDER BY rideTime MEASURES S.rideId as sRideId AFTER MATCH SKIP PAST LAST ROW PATTERN (S M{2,} E) DEFINE S AS S.isStart = true, M AS M.rideId <> S.rideId, E AS E.isStart = false AND E.rideId = S.rideId )
- 36. © 2018 data Artisans36 Todays processing landscape Streaming Batch
- 37. © 2018 data Artisans37 Batch/streaming unification
- 38. © 2018 data Artisans38 Into the Future Big state SQL
- 39. © 2018 data Artisans39 Thank s!