Apache Flink - a Gentle Start
- 1. A Gentle Start with Apache Flink Liangjun Jiang 10/21/2019
- 2. Me • Staff Software Engineer • Had two talks before (“Alibaba Technology”, “A Template for Enterprise App with Azure”) • Different technology stack (frontend, backend, some data infrastructure and engineering) • Contribute to UI automation framework (macaca) – the most popular and most feature ready cross platform automation tool • Has two kids • A softball (assistant) coach at Oak Hills Youth Association • Volunteering with Technology Education and Literacy in Schools (TEALS)
- 3. Outlines • Should we need to care stream processing? • Stream Processing technologies and challenges • Flink Features and Why Flink • Demo 0 – Flink Steam Word Count • Demo 1 – Flink SQL and Table API • Demo 2 – Flink in Real-time Recommendation • Online Resources • Questions
- 4. Stream processing – should you care? • Traffic info in Google Maps are getting more accurate? • Credit card charge fraud detection • E-commerce user online shopping tracking • Real time data warehouse • Server monitoring • Sensor (IoT) events • …
- 5. Should We Care? • Uber, Lyft, Google(Apple) Maps • Offline Data Warehouse to online (real- time) one to save money and provide more up-to-date information to decision maker • Real time recommendation for online shopping https://cwiki.apache.org/confluence/display/FLINK/Powered+by+Flink
- 6. Stream Processing technologies and challenges APACHE STORM SPARK STREAMING APACHE FLINK APACHE BEAM
- 7. The Challenges • You are taking subway and the train goes into a tunnel where has no cellular reception • Your stream processing (Flink, Spark, etc) job fails, server crashes, network not reachable • Your sinker (Hive, Cassandra, MySQL, etc) could fail • The end-2-end exactly once • Will the source resend a same message (Idempotent)
- 8. Flink
- 9. How Flink implements end-2-end exactly once • Checkpoint and Savepoint • Two-phase commit (leader election, pre- commit)
- 10. Flink Features and Performance • Time: Event time, processing time and ingestion time • Window: tumbling window, sliding window, session window • State (Queryable, broadcast) and state processors APIs • Table API and SQL • Batch (Dataset API) • Event Processing (CEP) • Operators (stream joining/splitting/side output, async I/O) • Deployment and Monitoring • Kerberos security model • …
- 12. Demo 1: a Basic Flink Stream Application: Word Count • Code • Build & Run with Socket Input • Submit Flink Job on Web UI: http://localhost:8081 • web UI features Image source: https://flink.apache.org/usecases.html > Stream processing with flink > Stream > processing > Stream processing > flink > (stream, 3) > (processing, 3) > (flink, 2) Use cases: 1. user puts item into the shopping cart then check out 2. add to cart but not checkout out 3. Temp. sensor measures temp. variation etc
- 13. Demo 2: NYC Taxi Demo: • SQL CLI client • SQL query on streaming • Window aggregate & non-window aggregate • Source is Kafka • Sinker is Kafka & ElasticSearch Stack: 1. Flink 2. ElasticSearch 3. Kafka 4. Zookeeper > docker-compose up –d - docker-compose exec sql-client ./sql- client.sh - Flink SQL> show tables; - Flink SQL > describe rides; - select * from Rides where isInNYC(lon, lat); // filter and show NYC rides Http://localhost:8081 SELECT psgCnt, COUNT(*) AS cnt FROM Rides WHERE isInNYC(lon, lat) GROUP BY psgCnt; // show rides with 1 passenger, 2 passengers, etc http://wuchong.me/blog/2019/08/20/flink-sql-training/#more
- 14. Demo 2: NYC Taxi – Cont’d Taxis in and out for each area in 5 mins SELECT toAreaId(lon, lat) AS area, TUMBLE_END(rideTime, INTERVAL '5' MINUTE) AS window_end, COUNT(*) AS cnt FROM Rides WHERE isInNYC(lon, lat) and isStart GROUP BY toAreaId(lon, lat), TUMBLE(rideTime, INTERVAL '5' MINUTE) HAVING COUNT(*) >= 5; //has been speeded up, will show data each 30 seconds INSERT INTO Sink_TenMinPsgCnts SELECT TUMBLE_START(rideTime, INTERVAL '10' MINUTE) AS cntStart, TUMBLE_END(rideTime, INTERVAL '10' MINUTE) AS cntEnd, CAST(SUM(psgCnt) AS BIGINT) AS cnt FROM Rides GROUP BY TUMBLE(rideTime, INTERVAL '10' MINUTE); Each 10 mins, count how many passengers are riding and publish to Kafka – will not demo Insert Query Result to ElasticSearch http://localhost:9200/area-cnts/_stats
- 15. Demo 3: a Real Time User Based Collaborative Filter-based Recommendation System Flink tasks – • Logs – computing raw data to get stats • Context – computing user’s behavior and timestamp for certain actions, • Profiling for user and product – user’s preference, product characteristic, user - product • collaborative filter based recommendation • top list – most clicked, favorited or viewed products https://github.com/CheckChe0803/flink-recommandSystem-demo 1. Hbase 2. Kafka + zookeeper 3. MySQL 4. Intelli J to run Flink tasks 5. Spring Boot to run Web UI
- 16. Demo 3: a Real Time User Based Collaborative Filter-based Recommendation System • Top products • Product Profiling and recommendation (profiling the top products based on user’s age, preference of color, product origin and style) • Collaborative filter based recommendation – based on the product item’s similarity to recommend the similar products to the user.
- 17. Online Resources • https://flink.apache.org/ • https://www.ververica.com • http://streamingsystems.net/ • My book: Flink In Practice focusing on examples is under the way
- 18. Questions