Flink Forward Berlin 2018: Stephan Ewen - Keynote: "Unlocking the next wave of applications with Stream Processing"
2 likes445 views
This document discusses a new technology called Streaming Ledger that provides ACID transaction guarantees for streaming data. Streaming Ledger allows applications to read and modify multiple data rows/keys within a transaction, sharing state between streams. It uses logical clocks to define a conflict-free processing schedule and handles key contention well. Streaming Ledger is built on Apache Flink and provides a new capability beyond Flink's existing exactly-once processing of single keys.
Flink Forward Berlin 2018: Stephan Ewen - Keynote: "Unlocking the next wave of applications with Stream Processing"
- 1. Stephan Ewen – data Artisans CTO, Apache Flink PMC UNLOCKING THE NEXT WAVE OF STREAMING APPLICATIONS
- 2. © 2018 data Artisans2 Streaming Applications over Time Real time Approximate Analytics Fraud Detection Online Machine Learning Data pipelines Realtime ETL Intrusion Prevention Financial Risk & Reporting Real time dashboards Anomaly Detection Logistics Tracking Recommender Systems Web Applications Masterdata Management Continuous Processing Continuous Processing and Analytics Unification of Analytics and Applications Data-driven Applications
- 3. © 2018 data Artisans3 Enablers of new Applications Abstractions, APIs Consistency Event-time, streaming SQL, state & time, CEP Exactly-once, savepoints Scalability high parallelism, large state Interoperability Deployment, Connectors, Operations
- 4. © 2018 data Artisans4 Enablers of new Applications Abstractions, APIs Consistency Event-time, streaming SQL, state & time, CEP Exactly-once, savepoints Interoperability Deployment, Connectors, Operations Steam SQL, Time-versioned Table Joins, SQL+CEP, … Framework and Library, REST-ified Flink, SQL Client, …Scalability high parallelism, large state Scalable timers, dynamic scaling, local recovery, …
- 5. © 2018 data Artisans5 Exactly-once Changed Applications Stateless Application K/V Store CRUD / request/response Applications Streaming Application State Stateful Stream Processing Applications … become …
- 6. © 2018 data Artisans6 Some Applications don't move to Stream Processing Application Relational Database
- 7. © 2018 data Artisans7 Limitation of Current Stream Processors Transferring money from one account (key) to another with transactional guarantees is not feasible The Limitation Example All stream processors so far can update a single key-at-a-time with correctness guarantees (exactly once)
- 8. © 2018 data Artisans8 Wouldn't it be great if Stream Processors could… • … access and update state with multiple keys at the same time • … maintain full isolation/correctness for the multi-key operations • … operate on multiple states at the same time • … share the states between multiple streams
- 9. © 2018 data Artisans9 TODAY WE ANNOUNCE The first system for serializable multi-row ACID transactions on streaming data
- 10. © 2018 data Artisans10
- 11. © 2018 data Artisans11 ‒ Atomicity: the transfer affects either both accounts or none ‒ Consistency: the transfer must only happen if the account have sufficient funds ‒ Isolation: no other operation can interfere and cause an incorrect result ‒ Durability: the result of the transfer is durable ACID Transactions for Multi-key Stream Processing Streaming Ledger provides ACID guarantees across multiple states, rows, and streams
- 12. © 2018 data Artisans12 The Evolution of Stream Processing accurate single-key applications Flink pioneered exactly once guarantees on true streaming: accurate analytics and applications, consistent single key at a time At-least-once guarantees Exactly-once guarantees approximate real time analytics No data loss but possible duplications to support real-time approximate analytics in a "speed layer" (lamda) ACID guarantees consistent general applications Streaming Ledger provides ACID guarantees supporting applications that read and modify several keys
- 13. © 2018 data Artisans13 Stateful Streaming vs. Streaming Ledger Storage Computing Single datum / key at a time Consistent total view data Artisans Streaming Ledger exactly-once stateful streaming Key/Value Stores Relational Databases
- 14. © 2018 data Artisans14 Tables, Streams, Transactions Account Balance Asset Qty. Deposit Function Transfer Function Xfer: A B: $ Swap: X Y Deposit: A: $ SUCCESS A=$, B=$ Transaction Event Streams Transaction Functions Result Streams
- 15. © 2018 data Artisans15 Example: Transferring Cash/Assets between Accounts
- 16. © 2018 data Artisans16 Example: Position-keeping, Reporting, Risk Management in Investment Banking
- 17. © 2018 data Artisans17 A Library on top of Apache Flink • No additional dependencies needed • Seamlessly integrates and composes with DataStream API and SQL • Read from- and write to all Flink connectors • Supports savepoints for upgrades
- 18. © 2018 data Artisans18 How does it work?
- 19. © 2018 data Artisans19 Relational Database embedded? No!
- 20. © 2018 data Artisans20 First-of-a-kind - Unique Approach Logical Clocks to define schedule event re-ordering & out-of-order processing Iterative streaming dataflows (0xf876ab78|0x0b7cc7a3)
- 21. © 2018 data Artisans21 Conflict-free Schedule Txn: (A,B,C) -> A Txn: (A,B,C) -> C Txn: (D,E) -> (D,F) transaction events define ordering for schedule reorder events to obey schedule B CA A D E D F A CB CA
- 22. © 2018 data Artisans22 Performance (early results) - Scalability (parallelism) 200 million rows 100% update queries 4 rows written/query
- 23. © 2018 data Artisans23 Performance (early results) – Key Contention Artificial case of extreme contention: 4 x 200,000 = 800,000 updates/sec on the same 1,000 keys Slowdown, but does not break like optimistic concurrency approaches 100% update queries 4 written/query
- 24. © 2018 data Artisans24 Running a distributed setup … and because we can… …we run a globally distributed setup
- 25. © 2018 data Artisans25
- 26. © 2018 data Artisans26 Apache Flink: The heavy lifter This technology is possible, because Apache Flink offers such powerful building blocks • Continuous processing • Iterative flows • Flexible state abstraction • Asynchronous checkpoints • Sophisticated event-time/watermarks
- 27. © 2018 data Artisans27 Part of data Artisans Platform River Edition Streaming Ledger is part of data Artisans Platform • Stream Edition: Apache Flink, Application Manager • River Edition: Apache Flink, Application Manager, Streaming Ledger API and single-node implementation under Apache 2.0 license
- 28. © 2018 data Artisans28 Learn more at At the booth
- 29. Thank you! Enjoy the conference!