Apache Flink
0 likes•78 views
Apache Flink is an open-source stream processing framework designed for high-volume, low-latency batch and stream processing, written in Java and Scala. It supports multiple data sources and provides powerful APIs for event processing, state management, and analytics, along with libraries for complex event processing and graph analysis. Flink is deployable on various cluster managers and is tailored for applications such as fraud detection, data analytics, and real-time data pipelines.
Apache Flink
- 1. What Is Apache Flink ? ● A stream processing framework ● Open source / Apache 2.0 license ● Written in Java and Scala ● For batch and stream processing ● For high volume , low latency ● Develop in Java, Scala, Python, SQL ● Automatic compilation/optimization into data flows
- 2. How Does Flink Work ? ● Process Unbounded and Bounded Data ● Uses file systems to consume/persistently store data i.e. – local, hadoop-compatible, Amazon S3, MapR FS, OpenStack Swift FS, Aliyun OSS and Azure Blob Storage ● Leverages In-Memory Performance ● Provides a rich function set for handling – Streams, state and time – When building applications ● Provides layered API's which provides a balance between – Conciseness and expressiveness – See next slide
- 3. How Does Flink Work ? Flink layered API's
- 4. Flink API's ● SQL & Table API ● DataStream API ● ProcessFunctions – event processing ● Flink also has libraries for common data processing – Complex Event Processing (CEP) – DataSet API – Gelly - library for scalable graph processing/analysis
- 6. Flink Deployment ● Deploy Flink to use the following cluster managers – YARN – Mesos – Kubernetes – Stand alone ● All application control communications via REST calls ● Deploy at any scale – multiple trillions of events per day – multiple terabytes of state – thousands of cores
- 8. Flink Stateful Functions ● Simplifies building distributed stateful applications ● Provides a runtime built for serverless architectures ● Key Benefits – Dynamic Messaging – Consistent State – Multi-language Support – No Database Required – Cloud Native – "Stateless" Operation
- 10. Flink Use Cases ● Event-driven Applications i.e. – Fraud detection – Anomaly detection ● Data Analytics Applications – Quality monitoring of Telco networks – Analysis of product updates & experiment evaluation in mobile applications ● Data Pipeline Applications – Real-time search index building in e-commerce – Continuous ETL in e-commerce
- 11. Flink Use Cases
- 12. Flink Use Cases
- 13. Available Books ● See “Big Data Made Easy” – Apress Jan 2015 ● See “Mastering Apache Spark” – Packt Oct 2015 ● See “Complete Guide to Open Source Big Data Stack – “Apress Jan 2018” ● Find the author on Amazon – www.amazon.com/Michael-Frampton/e/B00NIQDOOM/ ● Connect on LinkedIn – www.linkedin.com/in/mike-frampton-38563020
- 14. Connect ● Feel free to connect on LinkedIn – www.linkedin.com/in/mike-frampton-38563020 ● See my open source blog at – open-source-systems.blogspot.com/ ● I am always interested in – New technology – Opportunities – Technology based issues – Big data integration