Flink Forward Berlin 2018: Wei-Che (Tony) Wei - "Lessons learned from Migrating to a Stateful Streaming Framework"
0 likes320 views
The document discusses lessons learned from migrating to a stateful streaming framework at Appier, highlighting the transition from micro-batching to true streaming with Apache Flink. It includes key use cases, challenges faced during the migration, and valuable tips such as the importance of documentation, monitoring, and familiarization with the environment. Overall, it emphasizes that stateful streaming requires thoughtful design and is sensitive to its operational environment.
Flink Forward Berlin 2018: Wei-Che (Tony) Wei - "Lessons learned from Migrating to a Stateful Streaming Framework"
- 1. Lessons learnt from Migrating to a Stateful Streaming Framework Software Engineer , Appier Wei-Che(Tony) Wei
- 2. • Use case in Appier • Moving from micro batch to true streaming • Tips to conquer obstacles during the migration • Challenge from design in streaming way Outline © Appier Inc. All rights reserved.
- 3. • Use case in Appier • Moving from micro batch to true streaming • Tips to conquer obstacles during the migration • Challenge from design in streaming way Outline © Appier Inc. All rights reserved.
- 4. © Appier Inc. All rights reserved. About Appier Appier is a technology company which aims to provide artificial intelligence platforms to help enterprises solve their most challenging business problems. http://www.appier.com/en/index.html
- 5. © Appier Inc. All rights reserved. Use case ● Retargeting ● Fraud detection ● Real time recommendation ● ...
- 6. • Use case in Appier • Moving from micro batch to true streaming - Disadvantage from micro batch - New requirements - Our solution by using Flink • Tips to conquer obstacles during the migration • Challenge from design in streaming way Outline © Appier Inc. All rights reserved.
- 7. Dynamic Rule Service: Previous design © Appier Inc. All rights reserved.
- 8. • High latency • Struggle with back pressure • Hard to maintain states Disadvantage from micro batch © Appier Inc. All rights reserved.
- 9. • “Who visited this site twice in the past week” • “Who viewed this product in the site but didn’t put into cart” • “Who visited the certain pattern of some pages and bought this product” New requirements © Appier Inc. All rights reserved.
- 10. Dynamic Rule Service with Apache Flink © Appier Inc. All rights reserved.
- 11. Detail of JobGraph (Apache Flink 1.4.0 release) © Appier Inc. All rights reserved.
- 12. • Support more needs for stateful rules • Flexible architecture • Performance efficiency • Cost efficiency Improvements © Appier Inc. All rights reserved.
- 13. • Use case in Appier • Moving from micro batch to true streaming • Tips to conquer obstacles during the migration • Challenge from design in streaming way Outline © Appier Inc. All rights reserved.
- 14. Tip 1: Document and Mailing List are your best friends © Appier Inc. All rights reserved. Case Study: taskmanager.exit-on-fatal-akka-error: true https://ci.apache.org/projects/flink/flink-docs-release-1.4/ops/co nfig.html#jobmanager--taskmanager
- 15. Tip 2: Monitor and alert are important © Appier Inc. All rights reserved.
- 16. Tip 2: Monitor and alert are important © Appier Inc. All rights reserved.
- 17. Tip 2: Monitor and alert are important © Appier Inc. All rights reserved.
- 18. Tip 2: Monitor and alert are important © Appier Inc. All rights reserved.
- 19. Tip 2: Monitor and alert are important © Appier Inc. All rights reserved.
- 20. Tip 3: Be familiar with your environment and your job Case Study: checkpoint stuck with rocksdb statebackend and s3 filesystem © Appier Inc. All rights reserved.
- 21. Tip 3: Be familiar with your environment and your job Case Study: Verify bottleneck of our streaming job ⨯ network performance issue ✔ memory bound job met the resource limitation © Appier Inc. All rights reserved.
- 22. • Use case in Appier • Moving from micro batch to true streaming • Tips to conquer obstacles during the migration • Challenge from design in streaming way Outline © Appier Inc. All rights reserved.
- 23. • Replay expired data or rebuild states are complex • How to expose user-defined metrics well - Current matched count for each rule • Too expensive to use querible states • End-to-end verification is a tough job - How to verify those replay data is prepared Challenge from design in streaming way © Appier Inc. All rights reserved.
- 24. • Documents and information from mailing list help a lot. • Monitor and alert let you respond to problem and diagnose it quickly. • Stateful streaming is environment sensitive. Be careful of it. • Community is always behind you. TL;DR © Appier Inc. All rights reserved.
- 25. Thank you