Flink Forward San Francisco 2018 keynote: Stephan Ewen - "What turns stream processing from a tool into a platform?"
Download as PPTX, PDF•2 likes•1,112 views
The document outlines the evolution of stream processing architecture, emphasizing the shift from database-centric designs to applications built directly on data streams using Apache Flink. It discusses the role of Kubernetes in managing stateful applications and addresses challenges like data migration, stateful upgrades, and maintaining consistent application states. Additionally, it envisions the future of streaming data platforms, highlighting essential features such as role-based access control and support for batch processing.
Flink Forward San Francisco 2018 keynote: Stephan Ewen - "What turns stream processing from a tool into a platform?"
- 1. - STEPHAN EWEN, CO-FOUNDER & CTO STREAM PROCESSING FROM APPLICATIONS TO PLATFORMS
- 2. 2 A platform makes building new applications simple by taking care of the common and repeatable parts.
- 3. 3 Internal streaming data platforms built with Apache Flink
- 4. 4 Observation 1 Stream Processing is about building applications
- 5. Batch / Data Lake Architecture 5 a.k.a. collect now, figure out later
- 6. Streaming / Data-driven Applications 6 build applications directly on data streams
- 7. 7 Observation 2 Stream Processing changes the database-centric architecture
- 8. Recall last Flink Forward… 8 Classic tiered architecture Streaming architecture database layer compute layer application working state + historic state compute + stream storage and snapshot storage (backup) application state
- 9. Changing the Two Tier Architecture 9 reads/writes across tier boundary asynchronous writes of large blobs all modifications are local Classic tiered architecture Streaming architecture
- 13. Kubernetes & Stateful Applications 13 Kubernetes Database
- 14. What about stateful containers? 14 Kubernetes • Example: Scaling down a replicated database • 3 replicas, 4 node scale down need to move or reorganize data before container shutdown
- 15. Stateful Questions consistent stateful upgrades • application evolution and bug fixes migration of application state • cluster migration, A/B testing re-processing and reinstatement • fix corrupt results, bootstrap new applications state evolution (schema evolution) 15 A B
- 16. 16 Container-based Resource Orchestration Stateful Stream Processing & Snapshots Kubernetes Apache Flink Container-based platform for stateful data-driven applications dA Platform Code, Resource, Config, and Snapshot Management Application Manager
- 18. Versioned Applications, not Jobs/Jars 18 Stream Processing Application Version 3 Version 2 Version 1 Code and Application Snapshot upgrade upgrade New Application Version 3a Version 2a fork / duplicate
- 19. Architecture Apache Flink Stateful stream processing Kubernetes Container platform Logging Metrics dA Application Manager Application lifecycle management
- 20. 20 What could the future of a Streaming Data Platform look like?
- 21. The Usual Suspects Role-based access control Metadata management Cross Datacenter Failover / Disaster Recovery 21
- 22. Support for Batch Processing 22 Everything is a stream. Finite applications as a special case.
- 23. Periodic Bursty Stream Processing 23 time Bursty Event Stream (events only at end-of-day ) Checkpoint / Savepoint Store
- 24. Support a Broad Developer Audience 24 Streaming Data Platform …
- 25. Use Case Vertical Libraries 25 Streaming Data Platform SQL CEP … Machine Learning
- 26. Apache Flink Stateful stream processing Kubernetes Container platform Logging Metrics dA Application Manager Application lifecycle management dA Platform is a turnkey solution for stateful stream processing with Apache Flink.