Flink 1.0-slides
- 1. What’s new in Apache FlinkTM 1.0 Kostas Tzoumas @kostas_tzoumas
- 2. Flink 1.0 • March 8, 2016 • First release in 1.x.y series • Initiates backwards compatibility for selected APIs • More than 64 contributors • More than 450 JIRAs resolved 2
- 3. Flink 1.0: major features • Out of core state • Savepoints • CEP library • Improved monitoring & Kafka 0.9 support 3
- 4. Out of core state 4
- 5. Out of core state • Alternative to in-memory state • Powered by RocksDB instances in Flink TMs • Enabled by using the RocksDBStateBackend • State limited by disk space only • State checkpoints save RocksDB databases in reliable store 5
- 6. Savepoints 6
- 7. Production deployments • Maintaining stateful applications in production settings comes with its own challenges • Failures, code upgrades, cluster maintenance, … • Streaming jobs cannot be simply stopped and restarted 7
- 8. Reminder: fault tolerance • At least once, at most once, exactly once • Flink guarantees exactly-once processing • Flink guarantees end to end exactly-once with selected sources and sinks • e.g., Kafka —> Flink —> HDFS
- 9. How? Checkpoints • Flink guarantees fault tolerance by regularly taking checkpoints of the application state without ever stopping the execution • At failure, input stream is rewinded to the logical time of the last checkpoint 9
- 10. Introducing savepoints • A savepoint is a Flink checkpoint that (1) is taken by the user, (2) is accessible externally, and (3) never expires • Command line save & resume interface • Save: flink savepoint <JobID> • Resume: flink run -s <path/to/savepoint> <jobJar> 10
- 11. Savepoints and versions • A savepoint saves a version of a stateful application at a well-defined time • E.g.: take snapshots of one application at well-defined times 11
- 12. “Like git for state” • Branch off from savepoints creating a tree of running application versions 12
- 13. Essential for production deployments • Application code upgrades • Flink version upgrades • Maintenance, migration, debugging • What-if simulations • A/B testing • Time travel 13
- 15. FlinkCEP • What is Complex Event Processing? • A catch-all term • In our context: easily detect patterns in streams 15
- 16. 16
- 17. Pattern API 17
- 18. 18
- 19. 19
- 20. Other features in 1.0 • Support for Kafka 0.9 API (and hence MapR Streams) • Monitoring console: job submission, checkpoint statistics, detecting bottlenecks • See http://flink.apache.org/news/2016/03/08/release- 1.0.0.html 20
- 21. Closing 21
- 22. Summary • Flink 1.0: Initiating backwards compatibility and pushing the envelope even further for production streaming deployments 22
- 23. What’s next • SQL • Dynamic scaling (+ savepoints) • Hybrid in-memory/out-of-core state backend • Query-able state • Support for Apache Mesos • More connectors and sinks (Kinesis, Cassandra, …) 23
- 24. Join the community • Follow: @ApacheFlink, @dataArtisans • Read: flink.apache.org/blog, data-artisans.com/blog • Subscribe: (news | dev | user)@flink.apache.org