The Power of Distributed Snapshots in Apache Flink
3 likes1,002 views
The document discusses the power of stateful stream processing using Apache Flink, showcasing its capabilities for handling real-time and historic data efficiently. It highlights the architecture of Flink, emphasizing its ability to manage complex event streams, provide distributed snapshots, and ensure data consistency even in the face of failures. Furthermore, it addresses practical applications and user implementations, such as upgrades, rollbacks, and state management challenges in modern application development.
![Stateful Event & Stream Processing
12
Source
Transformation
Transformation
Sink
val lines: DataStream[String] = env.addSource(new FlinkKafkaConsumer09(…))
val events: DataStream[Event] = lines.map((line) => parse(line))
val stats: DataStream[Statistic] = stream
.keyBy("sensor")
.timeWindow(Time.seconds(5))
.sum(new MyAggregationFunction())
stats.addSink(new RollingSink(path))
Streaming
Dataflow
Source Transform Window
(state read/write)
Sink](https://image.slidesharecdn.com/untitled-180219101330/85/The-Power-of-Distributed-Snapshots-in-Apache-Flink-14-320.jpg)
![Powerful Abstractions
15
Process Function (events, state, time)
DataStream API (streams, windows)
Stream SQL / Tables (dynamic tables)
Stream- & Batch
Data Processing
High-level
Analytics API
Stateful Event-
Driven Applications
val stats = stream
.keyBy("sensor")
.timeWindow(Time.seconds(5))
.sum((a, b) -> a.add(b))
def processElement(event: MyEvent, ctx: Context, out: Collector[Result]) = {
// work with event and state
(event, state.value) match { … }
out.collect(…) // emit events
state.update(…) // modify state
// schedule a timer callback
ctx.timerService.registerEventTimeTimer(event.timestamp + 500)
}
Layered abstractions to
navigate simple to complex use cases](https://image.slidesharecdn.com/untitled-180219101330/85/The-Power-of-Distributed-Snapshots-in-Apache-Flink-17-320.jpg)
The Power of Distributed Snapshots in Apache Flink
- 1. The Power of Snapshots Stateful Stream Processing with Apache Flink Stephan Ewen QCon San Francisco, 2017 1
- 2. InfoQ.com: News & Community Site • Over 1,000,000 software developers, architects and CTOs read the site world- wide every month • 250,000 senior developers subscribe to our weekly newsletter • Published in 4 languages (English, Chinese, Japanese and Brazilian Portuguese) • Post content from our QCon conferences • 2 dedicated podcast channels: The InfoQ Podcast, with a focus on Architecture and The Engineering Culture Podcast, with a focus on building • 96 deep dives on innovative topics packed as downloadable emags and minibooks • Over 40 new content items per week Watch the video with slide synchronization on InfoQ.com! https://www.infoq.com/presentations/ distributed-stream-processing-flink
- 3. Purpose of QCon - to empower software development by facilitating the spread of knowledge and innovation Strategy - practitioner-driven conference designed for YOU: influencers of change and innovation in your teams - speakers and topics driving the evolution and innovation - connecting and catalyzing the influencers and innovators Highlights - attended by more than 12,000 delegates since 2007 - held in 9 cities worldwide Presented at QCon San Francisco www.qconsf.com
- 4. 2 Original creators of Apache Flink® dA Platform 2 Open Source Apache Flink + dA Application Manager
- 6. What changes faster? Data or Query? 4 Data changes slowly compared to fast changing queries ad-hoc queries, data exploration, ML training and (hyper) parameter tuning Batch Processing Use Case Data changes fast application logic is long-lived continuous applications, data pipelines, standing queries, anomaly detection, ML evaluation, … Stream Processing Use Case
- 10. Moving State into the Processors 8 Application External DBstate Stateless Stream Processor Stateful Stream Processor Application state
- 11. 9 Apache Flink
- 12. Apache Flink in a Nutshell 10 Queries Applications Devices etc. Database Stream File / Object Storage Stateful computations over streams real-time and historic fast, scalable, fault tolerant, in-memory, event time, large state, exactly-once Historic Data Streams Application
- 13. 11 Event Streams State (Event) Time Snapshots The Core Building Blocks real-time and hindsight complex business logic consistency with out-of-order data and late data forking / versioning / time-travel
- 14. Stateful Event & Stream Processing 12 Source Transformation Transformation Sink val lines: DataStream[String] = env.addSource(new FlinkKafkaConsumer09(…)) val events: DataStream[Event] = lines.map((line) => parse(line)) val stats: DataStream[Statistic] = stream .keyBy("sensor") .timeWindow(Time.seconds(5)) .sum(new MyAggregationFunction()) stats.addSink(new RollingSink(path)) Streaming Dataflow Source Transform Window (state read/write) Sink
- 15. Stateful Event & Stream Processing 13 Scalable embedded state Access at memory speed & scales with parallel operators
- 16. Event time and Processing Time 14 Event Producer Message Queue Flink Data Source Flink Window Operator partition 1 partition 2 Event Time Ingestion Time Processing Time Broker Time Event time, Watermarks, as in the Dataflow model
- 17. Powerful Abstractions 15 Process Function (events, state, time) DataStream API (streams, windows) Stream SQL / Tables (dynamic tables) Stream- & Batch Data Processing High-level Analytics API Stateful Event- Driven Applications val stats = stream .keyBy("sensor") .timeWindow(Time.seconds(5)) .sum((a, b) -> a.add(b)) def processElement(event: MyEvent, ctx: Context, out: Collector[Result]) = { // work with event and state (event, state.value) match { … } out.collect(…) // emit events state.update(…) // modify state // schedule a timer callback ctx.timerService.registerEventTimeTimer(event.timestamp + 500) } Layered abstractions to navigate simple to complex use cases
- 19. Event Sourcing + Memory Image 17 event log persists events (temporarily) event / command Process main memory update local variables/structures periodically snapshot the memory
- 20. Event Sourcing + Memory Image 18 Recovery: Restore snapshot and replay events since snapshot event log persists events (temporarily) Process
- 21. Consistent Distributed Snapshots 19 Scalable embedded state Access at memory speed & scales with parallel operators
- 23. Consistent Distributed Snapshots 21 Trigger checkpoint Inject checkpoint barrier
- 24. Consistent Distributed Snapshots 22 Take state snapshot Trigger state copy-on-write
- 25. Consistent Distributed Snapshots 23 Persist state snapshots Persist snapshots asynchronously Processing pipeline continues
- 26. Consistent Distributed Snapshots 25 Re-load state Reset positions in input streams Rolling back computation Re-processing
- 27. Consistent Distributed Snapshots 26 Restore to different programs
- 28. 27 Checkpoints and Savepoints in Apache Flink
- 29. Speed or Operability? 28 Fast snapshots Checkpoint Flexible Operations on Snapshots Savepoint What to optimize for?
- 30. Savepoints: Opt. for Operability Self contained: No references to other checkpoints Canonical format: Switch between state structures Efficiently re-scalable: Indexed by key group Future: More self-describing serialization format for to archiving / versioning (like Avro, Thrift, etc.) 29
- 31. Checkpoints: Opt. for Efficiency Often incremental: • Snapshot only diff from last snapshot • Reference older snapshots, compaction over time Format specific to state backend: • No extra copied or re-encoding • Not possible to switch to another state backend between checkpoints Compact serialization: Optimized for speed/space, not long term archival and evolution Key goups not indexed: Re-distribution may be more expensive 30
- 32. 31 What else are snapshots / checkpoints good for?
- 33. What users built on checkpoints Upgrades and Rollbacks Cross Datacenter Failover State Archiving State Bootstrapping Application Migration Spot Instance Region Arbitrage A/B testing … 32
- 34. 33 Distributed Snapshots and side effects
- 35. Transaction coordination for side fx 34 One snapshot can transactionally move data between different systems Snapshots may include side effects
- 36. Transaction coordination for side fx Similar to a distributed 2-phase commit Coordinated by asynchronous checkpoints, no voting delays Basic algorithm: • Between checkpoints: Produce into transaction or Write Ahead Log • On operator snapshot: Flush local transaction (vote-to-commit) • On checkpoint complete: Commit transactions (commit) • On recovery: check and commit any pending transactions 35
- 37. 36 Distributed Snapshots and Application Architectures (A Philosophical Monologue)
- 38. Good old centralized architecture 37 The big mean central database $$$ The grumpy DBA Application Application Application Application
- 39. Stateful Stream Proc. & Applications 38 Application Application Application Application Application decentralized infrastructure DevOps decentralized responsibilities still involves managing databases
- 40. Stateless Application Containers 39 State management is nasty, let's pretend we don't have to do it
- 41. Stateless Application Containers 40 Kudos to Kiki Carter for the Broccoli Metaphor Broccoli (state management) is nasty, let's pretend we don't have to eat do it
- 42. Stateful Stream Proc. to the rescue 41 Application Sensor APIs Application Application Application very simple: state is just part of the application
- 43. Compute, State, and Storage 42 Classic tiered architecture Streaming architecture database layer compute layer application state + backup compute + stream storage and snapshot storage (backup) application state
- 44. Performance 43 synchronous reads/writes across tier boundary asynchronous writes of large blobs all modifications are local Classic tiered architecture Streaming architecture
- 45. Consistency 44 distributed transactions at scale typically at-most / at-least once exactly once per state =1 =1 Classic tiered architecture Streaming architecture
- 46. Scaling a Service 45 separately provision additional database capacity provision compute and state together Classic tiered architecture Streaming architecture provision compute
- 47. Rolling out a new Service 46 provision a new database (or add capacity to an existing one) simply occupies some additional backup space Classic tiered architecture Streaming architecture provision compute and state together
- 48. Time, Completeness, Out-of-order 47 ? event time clocks define data completeness event time timers handle actions for out-of-order data Classic tiered architecture Streaming architecture
- 49. Stateful Stream Processing 48 Application Sensor APIs Application Application Application very simple: state is just part of the application
- 50. The Challenges with that: Upgrades are stateful, need consistency • application evolution and bug fixes Migration of application state • cluster migration, A/B testing Re-processing and reinstatement • fix corrupt results, bootstrap new applications State evolution (schema evolution) 49
- 51. 50 Consistent Distributed Snapshots The answer (my personal and obviously biased take)
- 53. 52 Thank you very much (shameless plug)
- 55. Appendix 54
- 56. 55 Details about Snapshots and Transactional Side Effects
- 57. Exactly-once via Transactions 56 chk-1 chk-2 TXN-1 ✔chk-1 ✔chk-2 TXN-2 ✘ TXN-3 Side effect ✔ global ✔ global
- 58. Transaction fails after local snapshot 57 chk-1 chk-2 TXN-1 ✔chk-1 TXN-2 ✘ TXN-3 ✔ global Side effect
- 59. Transaction fails before commit… 58 chk-1 chk-2 TXN-1 ✔chk-1 TXN-2 ✘ TXN-3 ✔ global ✔ global Side effect
- 60. … commit on recovery 59 chk-2 TXN-2 TXN-3 ✔ global recover TXN handle chk-3 Side effect
- 61. Watch the video with slide synchronization on InfoQ.com! https://www.infoq.com/presentations/ distributed-stream-processing-flink