Flink Forward Berlin 2018: Shriya Arora - "Taming large-state to join datasets for Personalization"
0 likes1,134 views
This talk focuses on managing large-scale state for personalization using streaming solutions, specifically through Flink's state management for real-time event processing. It discusses challenges such as handling high-volume input streams, late-arriving events, and the need for efficient model training on fresh data. Future work includes state migration and improving operational visibility.
Flink Forward Berlin 2018: Shriya Arora - "Taming large-state to join datasets for Personalization"
- 1. Shriya Arora Data Engineering & Infrastructure Taming large state for Personalization
- 2. What is this talk about? ● Deriving signal from high volume real-time events ● Using Flink State management to achieve real-time join ● Operations and path to production ● Challenges and Learnings
- 3. What are Merched Impressions?
- 4. What is a Play-Impression Takerate? ● Number of merched impressions per user play ● Attributes of impressions leading to the play ● Attributes of the play coming from different impressions
- 5. What do we use it for? ● Ranking Videos ● Targeting and Reach ● Content Promotion ● Asset Personalization
- 6. Volume and Scale: ● 130M members ● ~10B Impressions ● ~2.5B Play Events ● 140M Play hours/day
- 7. Why do we need a streaming solution for take-rate ● Model Training on fresher data ○ Reduce time delay between event generation and signal ○ Faster feedback around launches ○ Events relevance temporal in nature ● Long turnaround time on error correction ○ Long running batch jobs have all-or-none failure modes ○ Lack of Real-time auditing delays error-detection
- 8. What are the challenges we will need to solve ? ● High-volume input streams ● Out-of-order and late-arriving events ● Large State ○ ~1TB State/ region
- 9. Approaches: #1 Window Joins ○ Events are delayed independent of each other #2 Aggregation over Windows followed by Join ○ Stream can be reduced as they are held in state
- 10. Approaches: #3 CoProcess Function with Single MapState ○ High variance in stream volumes and logic #4 CoProcess Function with two Value states ○ Each stream gets its own value state
- 11. A tale of two states ● CoProcess Function ● Save each Keyed stream into its own ValueState ● For each event in stream, reduce state on duplicates ● For each event in either stream, cross query across states ● Use timerService to expire events from State
- 12. Data Flow Architecture Play stream Impressions stream State 1 State 2 F(x) + Ts F(y) + Ts Co-process Fn Output keyBy
- 13. Anatomy of CoProcess Function def processElement1{value: T, ctx:Context ..} Access elements of the first stream, update and reduce state, lookup state 2 for out-of-order joins, apply timer def processElement2{value: K, ctx:Context ..} Access elements of the second stream, lookup and join to state 1, apply timer def onTimer{ts: Long ...} Clear up state based on event time ts.
- 14. State management
- 15. A tale of two states
- 16. Challenges with Operations Visibility into application event time progression ○ Flink UI bug: FLINK-8949
- 17. Challenges with Operations cont.. ● Visibility into State size ○ RocksDB Statistics have to be logged manually
- 18. Future Work ● State migration ● Data restatement and recovery