Apache Flink: Real-World Use Cases for Streaming Analytics
Download as PPTX, PDF28 likes12,573 views
The document discusses Apache Flink, emphasizing its transition from batch to streaming analytics, which is driven by the need for real-time data processing in various industries. It highlights key differentiators of Flink, such as low latency, high throughput, and its ability to ensure accurate results even during failures. Additionally, it explores real-world use cases across sectors like finance, healthcare, and retail, illustrating the growing adoption of streaming analytics for competitive advantage.
Apache Flink: Real-World Use Cases for Streaming Analytics
- 1. Apache Flink: Real-World Use Cases for Streaming Analytics Slim Baltagi @SlimBaltagi Brazil - Sao Paulo Apache Flink Meetup March 17th, 2016
- 2. Agenda I. What is Apache Flink Stack? II. Movement from Batch Analytics to Streaming Analytics III. Key Differentiators of Apache Flink for Streaming Analytics IV. Real-World Use Cases with Flink for Streaming Analytics V. Who is using Flink? VI. Where do you go from here? 2
- 3. I. What is Apache Flink stack? Gelly Table HadoopM/R SAMOA DataSet (Java/Scala/Python) Batch Processing DataStream (Java/Scala) Stream Processing FlinkML Local Single JVM Embedded Docker Cluster Standalone YARN, Mesos (WIP) Cloud Google’s GCE Amazon’s EC2 IBM Docker Cloud, … ApacheBeam ApacheBeam MRQL Table Cascading Runtime : Distributed Streaming Dataflow Zeppelin DEPLOYSYSTEMAPIs&LIBRARIESSTORAGE Files Local HDFS S3, Azure Storage Tachyon Databases MongoDB HBase SQL … Streams Flume kafka RabbitMQ … Batch Optimizer Stream Builder Storm FlinkCEP Gelly-Stream 3
- 4. I. What is Apache Flink stack? See First Apache Flink meetup in South America that I gave as a webinar on February 24th 2016. It is titled: Introduction to Apache Flink: What, How, Why, Who, Where? https://www.youtube.com/watch?v=YAKdD1rHCxs (Part 1) See similar talk on February 2nd 2016 that I previously gave a at the New York City Apache Flink which. Now, the world’s largest Flink meetup • Slideshttp://www.slideshare.net/sbaltagi/apacheflinkwhathowwhywhowhe rebyslimbaltagi-57825047 • Video recording https://www.youtube.com/watch?v=G77m6Ou_kFA Flink Knowledge Base: all resources related to Flink http://sparkbigdata.com/component/tags/tag/27-flink 4
- 5. Agenda I. What is Apache Flink Stack? II. Movement from Batch Analytics to Streaming Analytics III. Key Differentiators of Apache Flink for Streaming Analytics IV. Real-World Use Cases with Flink for Streaming Analytics V. Who is using Flink? VI. Where do you go from here? 5
- 6. II. Movement from Batch Analytics to Streaming Analytics Batch Streaming High-latency apps Low-latency apps Static Files Event Streams Process-after-store Sense-and-respond Batch processors Stream processors 6
- 7. What is batch processing? Many big data sources represent series of events that are continuously produced. Example: tweets, web logs, user transactions, system logs, sensor networks, … Batch processing: These events are collected together based on the number of records or a certain period of time (a day for example) and stored somewhere to be processed as a finite data set. What’s the problem with ‘process-after-store’ model: • Unnecessary latencies between data generation and analysis & actions on the data. • Implicit assumption that the data is complete after a given period of time and can be used to make accurate predictions for example. 7
- 8. What is stream processing? Most data is available as series of events (click streams, mobile apps data, .. ) continuously produced by a variety of applications and systems in the enterprise. Data sources are not anymore typical enterprise sources but new ones such as social media data, sensor data … Data from disparate systems (internally and externally) can be integrated in a central hub and: Made available as low-latency data streams required for real-time stream processing. Loaded into your data warehouse for offline analysis. 8
- 9. Factors behind the movement from Batch Analytics to Streaming Analytics There is a movement in Big Data processing from Batch Analytics to Streaming Analytics driven by many factors: • Data streams: Sensors networks, mobile apps data, .. • Technology: Rapidly growing open source streaming analytics tools, vendors innovating in this space, more mobile devices than human beings, cloud services for real-time stream processing… • Business: Organizations are more and more embracing streaming analytics for faster time to insight and competitive advantages. • Customers: Costumers are becoming more and more demanding for instant responses in the way they are used to in social networks: twitter, facebook, linkedin… 9
- 10. Agenda I. What is Apache Flink Stack? II. Batch vs. Streaming Analytics III. Key Differentiators of Apache Flink for Streaming Analytics IV. Real-World Use Cases with Flink for Streaming Analytics V. Who is using Flink? VI. Where do you go from here? 10
- 11. III. Key Differentiators of Apache Flink for Streaming Analytics The 8 Requirements of Real-Time Stream Processing, Stonebraker et al. 2005 • Original paper http://cs.brown.edu/~ugur/8rulesSigRec.pdf • A short summaryhttp://blog.acolyer.org/2014/12/03/the-8-requirements-of- real-time-stream-processing/ Apache Flink fulfills all these requirements and more! • http://data-artisans.com/real-time-stream-processing-the-next-step-for-apache-flink/ • http://data-artisans.com/flink-0-10-a-significant-step-forward-in-open-source-stream- processing/ • http://data-artisans.com/flink-1-0-0/ • https://cloud.google.com/dataflow/blog/dataflow-beam-and-spark-comparison • https://docs.google.com/document/d/1ExmtVpeVVT3TIhO1JoBpC5JKXm- 778DAD7eqw5GANwE/edit • http://www.slideshare.net/robertmetzger1/january-2016-flink-community-update-roadmap- 2016/9 11
- 12. III. Key Differentiators of Apache Flink for Streaming Analytics True Low latency streaming engine: fast results in milliseconds High throughput: handle large data amounts (millions of events per second) • http://data-artisans.com/extending-the-yahoo-streaming-benchmark/ Exactly once guarantees: Correct results, also in failure cases • http://data-artisans.com/high-throughput-low-latency-and-exactly-once-stream- processing-with-apache-flink/ Programmability: Higher level, Intuitive and easy to use APIs Backpressure refers to the situation where a system is receiving data at a higher rate than it can process during a temporary load spike. • http://data-artisans.com/how-flink-handles-backpressure/ Event time and out of order stream processing • http://data-artisans.com/how-apache-flink-enables-new-streaming-applications- part-1/ Stateful stream processing and versioning state • http://data-artisans.com/how-apache-flink-enables-new-streaming-applications/ 12
- 13. Agenda I. What is Apache Flink Stack? II. Batch vs. Streaming Analytics III. Key Differentiators of Apache Flink for Streaming Analytics IV. Real-World Use Cases with Flink for Streaming Analytics V. Who is using Flink? VI. Where do you go from here? 13
- 14. IV. Real-World Use Cases with Flink for Streaming Analytics Stonebraker et al. make the case in 2005 that stream processing is going to become increasingly important. Not just for the usual finance, fraud, and command-and- control use cases, but also….… “as the “sea change” caused by cheap micro-sensor technology takes hold, we expect to see everything of material significance on the planet get “sensor-tagged” and report its state or location in real time. This sensorization of the real world will lead to a “green field” of novel monitoring and control applications with high-volume and low- latency processing requirements.” Reference:http://blog.acolyer.org/2014/12/03/the-8-requirements-of-real-time- stream-processing/ 14
- 15. Shift from Reactive approach to proactive approach Capturing new data and providing the ability to process streams of this data is allowing organizations to shift • From: taking a REACTIVE, post transaction approach • To: more of a PROACTIVE, pre decision approach to interactions with their customers, suppliers and employees. Again, no matter the vertical, this transition is happening. 15
- 16. …to real-time personalization From static branding …to repair before break From break then fix …to designer medicine From mass treatment …to automated algorithms From educated investing …to 1x1 targeting From mass branding A shift in Advertising A shift in Financial Services A shift in Healthcare A shift in Retail A shift in Manufacturing Big Data Analytics Frameworks enable shifting the business from… Reactive Proactive Shift from Reactive approach to proactive approach 16
- 17. Real-Time Monitoring of Customer Activity Events 17
- 18. Generic Streaming Analytics Architectural pattern.Event Producers EventCollector EventBroker EventProcessor Indexer Visualizer/Search • Kafka • RabitMQ • JMS • Flink • Spark • Storm • Samza • ElasticSearch • Solr • Cassandra • NoSQL DB • Kibana • Custom GUI • Flume • SpringXD • Logstash • Nifi • Fluentd • Apps • Devices • Sensors 18
- 19. IV. Real-World Use Cases with Flink for Streaming Analytics Below is list several use cases, taken from real industrial situations: Financial Services – Real-time fraud detection. – Real-time mobile notifications. Healthcare – Smart hospitals - collect data and readings from hospital devices (vitals, IVs, MRI, etc.) and analyze and alert in real time. – Biometrics - collect and analyze data from patient devices that collect vitals while outside of care facilities. Ad Tech – Real-time user targeting based on segment and preferences. Oil & Gas • Real-time monitoring of pumps/rigs. 19
- 20. IV. Real-World Use Cases with Flink for Streaming Analytics Retail • Build an intelligent supply chain by placing sensors or RFID tags on items to alert if items aren’t in the right place, or proactively order more if supply is low. • Smart logistics with real-time end-to-end tracking of delivery trucks. Telecommunications • Real-time antenna optimization based on user location data. • Real-time charging and billing based on customer usage, ability to populate up-to-date usage dashboards for users. • Mobile offers. • Optimized advertising for video/audio content based on what users are consuming. 20
- 21. Agenda I. What is Apache Flink Stack? II. Batch vs. Streaming Analytics III. Key Differentiators of Apache Flink for Streaming Analytics IV. Real-World Use Cases with Flink for Streaming Analytics V. Who is using Flink? VI. Where do you go from here? 21
- 22. V. Who is using Flink? ho is using Apache Flink?How companies are using Flink as presented at Flink Forward 2015. Kostas Tzoumas and Stephan Ewen. http://www.slideshare.net/stephanewen1/flink-use-cases-bay-area-meetup- october-2015 Powered by Flink page: https://cwiki.apache.org/confluence/display/FLINK/Powered+by+Flink 22
- 23. V. Who is using Flink? is using Apache Flink? has its hack week and the winner was a Flink based streaming project! December 18, 2015 • Extending the Yahoo! Streaming Benchmark and Winning Twitter Hack-Week with Apache Flink. Posted on February 2, 2016 by Jamie Grier http://data-artisans.com/extending-the- yahoo-streaming-benchmark/ did some benchmarks to compare performance of their use case implemented on Apache Storm against Spark Streaming and Flink. Results posted on December 18, 2015 • http://yahooeng.tumblr.com/post/135321837876/benchmarking- streaming-computation-engines-at • http://data-artisans.com/extending-the-yahoo-streaming-benchmark/ • https://github.com/dataArtisans/yahoo-streaming-benchmark 23
- 24. Agenda I. What is Apache Flink Stack? II. Batch vs. Streaming Analytics III. Key Differentiators of Apache Flink for Streaming Analytics IV. Real-World Use Cases with Flink for Streaming Analytics V. Who is using Flink? VI. Where do you go from here? 24
- 25. VI. Where do you go from here? A few resources for you: • Flink at the Apache Software Foundation: flink.apache.org/ • Free ebook from MapR: Streaming Architecture: New Designs Using Apache Kafka and MapR Streams https://www.mapr.com/streaming-architecture-using-apache-kafka-mapr- streams • Free Apache Flink training from data Artisans http://dataartisans.github.io/flink-training/ Still version 0.10.1 and not latest 1.0 • Flink Knowledge Base: One-Stop for everything related to Apache Flink http://sparkbigdata.com/component/tags/tag/27-flink • Apache Flink in Action is probably the First book on Apache Flink! It will be published by Manning. I am co- authoring this book! Please stay tuned for the MEAP: Manning Early Access Program! 25
- 26. VI. Where do you go from here? A few takeaways : • Organizations are more and more embracing streaming analytics for: • Use cases requiring lower latency: monitoring, altering, … • Faster time to insight • Competitive advantages • By leveraging streaming analytics, new startups are challenging established companies. Example: Pay-As-You-Go insurance or Usage-Based Auto Insurance • Speed is said to have become the new currency of business. 26
- 27. Thanks! To all of you for attending! Let’s keep in touch! • [email protected] • @SlimBaltagi • https://www.linkedin.com/in/slimbaltagi Any questions? 27