Real Time Visibility with Flink
Download as PPTX, PDF0 likes95 views
The document outlines a vision for a competitive mobility marketplace that enhances urban mobility efficiency through real-time data integration and processing using Apache Flink. It details the challenges of handling complex and scattered data and explains the concept of data fusion to provide better analytics and user experiences. The presentation also covers technical implementations, including streaming architecture and integration with existing systems for improved data management.
![Union + keyBy + ProcessFunction
dA Platform Streaming Ledger SDK & CoGroupedStreams[1] inspired from the
• FULL JOIN behavior
• Event emitted per update (N + M)
• Good for simple JOIN logic
• Requires extra work](https://image.slidesharecdn.com/flinkrealtimevisibility-190515065951/85/Real-Time-Visibility-with-Flink-18-320.jpg)
Real Time Visibility with Flink
- 1. Realtime Visibility with Flink Rafi Aroch | @RafiAroch Data Architect
- 2. Create a competitive mobility market, in order to achieve urban mobility efficiency OurVision: Become the Marketplace of Mobility
- 3. Demand Supply Hotel Airlines Private Hire Airplane Bicycle Retail Healthcare End-Users Car Rental Bus Train TaxiMarketplace Consumer Application B2C SDK B2D Partners B2B2C Our Solution: The One-Stop-Shop Mobility Marketplace
- 6. Requirements BI Support Data Scientists real-time view of what’s going on build models offline & make decisions in real-time easy-to-work-with data to avoid complex ETLs
- 7. Limitations • Processing is batch based • Data freshness limited to 1-2 hours • Raw data was too complex to work with • Data scattered in many tables requires many ETLs & JOINS
- 9. We needed higher quality data, served in low latency
- 10. Data Fusion Data fusion is the process of integrating multiple data sources to produce more consistent, accurate, and useful information than that provided by any individual data source. Wikipedia
- 11. Meet Apache Flink Apache Flink® - Stateful Computations over Data Streams
- 13. Ride Fusion - Result • Processing is done in stream, on every event • fused-ride • Topic is getting published on every update • Events go back to the Datalake for analytical use • fused-ride-agg • Topic is updated every 1 min with active rides • Events are inserted into Postgres for user-facing applications • Data is better arranged, flattened, transformed • Many RAW topics are fused into one topic. All required data should be contained in the fused-ride event, without JOINs • Prediction services can evaluate and respond in real-time
- 15. Ride Fusion
- 16. SessionWindows s
- 17. SessionWindow JOIN • INNER JOIN – may not emit any results! • Pairwise combinations are passed on to the JoinFunction (N * M) • Enables custom JOIN logic
- 18. Union + keyBy + ProcessFunction dA Platform Streaming Ledger SDK & CoGroupedStreams[1] inspired from the • FULL JOIN behavior • Event emitted per update (N + M) • Good for simple JOIN logic • Requires extra work
- 19. Protobuf • Protobuf is not natively supported • Only custom Kryo serializer is an option • Performance hit • No state migration • Implemented ProtoTypeInfo + ProtoTypeSerializer • Need to specify TypeInformation where needed to avoid Kryo fallback • Planned to be added in Flink 1.9 - FLINK-11333
- 20. Unified Batch & Stream • We wanted the ability to re-use the same streaming code for batch. Used for backfill from/to S3 • Unified Batch & Stream is not straight-forward (yet) • StreamingFileSink does not support: • Bounded streams • Protobuf to Parquet writer • Using the “older” BucketingSink with custom Parquet writer • Getting a lot of focus for next releases - FLINK-11875
- 21. Dockerized Flink on ECS • No online resources (Only K8S, compose & standalone) • Binding to a host from within Docker • Port collisions when multiple containers sit on the same host • Dynamic port-mapping • Using custom entrypoint script for host and port setup • Jenkins pipeline + Terraform for CI/CD
- 22. Q&A
Editor's Notes
- #5: add bikes / train
- #6: There are different products & multiple kafka topics each
- #7: We have 3 main stakeholders They have typical requirements from their data
- #8: What we’re the limitations with our current architecture?
- #9: It’s challenging to do BI like that
- #10: We came to the conclusion…
- #11: We needed to take our raw data and fuse it into something better
- #12: A few words on Flink to whoever is hearing about Flink for the first time
- #13: We added dockerized Flink on ECS Consuming from the RAW topics, fusing the events into a fused-ride event Fused-ride-agg is a big aggregated event containing all updates so far
- #14: what were we able to achieve
- #15: add custom trigger when "close event" arrives + 5h inactivity check if aggregate cleans up after 1 minut (make process funstion with state) maybe add a join explanation chart maybe add chart from raw events to fused event
- #16: what were we able to achieve
- #17: Let's go over some of the main concepts required in this job
- #18: Let’s look at 2 approaches to implement Data Fusion with Flink First one is by using Session Window JOIN
- #19: add custom trigger when "close event" arrives + 5h inactivity check if aggregate cleans up after 1 minut (make process funstion with state) maybe add a join explanation chart maybe add chart from raw events to fused event