ApacheCon: Apache Flink - Fast and Reliable Large-Scale Data Processing
Download as PPTX, PDF2 likes1,492 views
This document provides an overview of Apache Flink, a distributed dataflow processing system for large-scale data analytics. Flink supports both stream and batch processing with easy to use APIs in Java and Scala. It focuses on fast and reliable processing at large scales and includes libraries for machine learning, graphs, and SQL-like queries.
![Counting Words in Batch and Stream
12
case class WordCount (word: String, count: Int)
val lines: DataStream[String] = env.fromSocketStream(...)
lines.flatMap {line => line.split(" ")
.map(word => WordCount(word,1))}
.window(Count.of(1000)).every(Count.of(100))
.groupBy("word").sum("count")
.print()
val lines: DataSet[String] = env.readTextFile(...)
lines.flatMap {line => line.split(" ")
.map(word => WordCount(word,1))}
.groupBy("word").sum("count")
.print()
DataSet API (batch):
DataStream API (streaming):](https://image.slidesharecdn.com/flink-apachecon-150416143023-conversion-gate01/85/ApacheCon-Apache-Flink-Fast-and-Reliable-Large-Scale-Data-Processing-12-320.jpg)
![Data Flow Optimizer
DataSource
orders.tbl
Filter DataSource
lineitem.tbl
Join
Hybrid Hash
buildHT probe
broadcast forward
Combine
Reduce
sort[0,1]
DataSource
orders.tbl
Filter DataSource
lineitem.tbl
Join
Hybrid Hash
buildHT probe
hash-part [0] hash-part [0]
hash-part [0,1]
Reduce
sort[0,1]
Best plan
depends on
relative sizes of
input filespartial sort[0,1]
39](https://image.slidesharecdn.com/flink-apachecon-150416143023-conversion-gate01/85/ApacheCon-Apache-Flink-Fast-and-Reliable-Large-Scale-Data-Processing-39-320.jpg)
ApacheCon: Apache Flink - Fast and Reliable Large-Scale Data Processing
- 1. Apache Flink Fast and Reliable Large-Scale Data Processing Fabian Hueske [email protected] @fhueske 1
- 2. About me Fabian Hueske [email protected] • PMC member of Apache Flink • With Flink since its beginnings in 2009 – back then an academic research project called Stratosphere
- 3. What is Apache Flink? Distributed Data Flow Processing System • Focused on large-scale data analytics • Real-time stream and batch processing • Easy and powerful APIs (Java / Scala) • Robust execution backend 3
- 4. 4 Flink in the Hadoop Ecosystem TableAPI GellyLibrary MLLibrary ApacheSAMOA Optimizer DataSet API (Java/Scala) DataStream API (Java/Scala) Stream Builder Runtime Local Cluster Yarn Apache TezEmbedded ApacheMRQL Dataflow HDFS S3JDBCHCatalog Apache HBase Apache Kafka Apache Flume RabbitMQ Hadoop IO ... Data Sources Execution Environments Flink Core Libraries TableAPI
- 5. What is Flink good at? It‘s a general-purpose data analytics system • Complex and heavy ETL jobs • Real-time stream processing with flexible windows • Analyzing huge graphs • Machine-learning on large data sets • ... 5
- 6. Flink in the ASF • Flink entered the ASF about one year ago – 04/2014: Incubation – 12/2014: Graduation • Strongly growing community – 17 committers – 15 PMC members 0 20 40 60 80 100 120 Nov-10 Apr-12 Aug-13 Dec-14 #unique git committers (w/o manual de-dup) 6
- 7. Where is Flink moving? A "use-case complete" framework to unify batch & stream processing Data Streams • Kafka • RabbitMQ • ... “Historic” data • HDFS • JDBC • ... Analytical Workloads • ETL • Relational processing • Graph analysis • Machine learning • Streaming data analysis 7 Goal: Treat batch as finite stream
- 8. HOW TO USE FLINK? Programming Model & APIs 8
- 9. DataSets and Transformations ExecutionEnvironment env = ExecutionEnvironment.getExecutionEnvironment(); DataSet<String> input = env.readTextFile(input); DataSet<String> first = input .filter (str -> str.contains(“Apache Flink“)); DataSet<String> second = first .map(str -> str.toLowerCase()); second.print(); env.execute(); Input First Secondfilter map 9
- 10. Expressive Transformations • Element-wise – map, flatMap, filter, project • Group-wise – groupBy, reduce, reduceGroup, combineGroup, mapPartition, aggregate, distinct • Binary – join, coGroup, union, cross • Iterations – iterate, iterateDelta • Physical re-organization – rebalance, partitionByHash, sortPartition • Streaming – window, windowMap, coMap, ... 10
- 11. Rich Type System • Use any Java/Scala classes as a data type – Special support for tuples, POJOs, and case classes – Not restricted to key-value pairs • Define (composite) keys directly on data types – Expression – Tuple position – Selector function 11
- 12. Counting Words in Batch and Stream 12 case class WordCount (word: String, count: Int) val lines: DataStream[String] = env.fromSocketStream(...) lines.flatMap {line => line.split(" ") .map(word => WordCount(word,1))} .window(Count.of(1000)).every(Count.of(100)) .groupBy("word").sum("count") .print() val lines: DataSet[String] = env.readTextFile(...) lines.flatMap {line => line.split(" ") .map(word => WordCount(word,1))} .groupBy("word").sum("count") .print() DataSet API (batch): DataStream API (streaming):
- 13. Table API • Execute SQL-like expressions on table data – Tight integration with Java and Scala APIs – Available for batch and streaming programs val orders = env.readCsvFile(…) .as('oId, 'oDate, 'shipPrio) .filter('shipPrio === 5) val items = orders .join(lineitems).where('oId === 'id) .select('oId, 'oDate, 'shipPrio, 'extdPrice * (Literal(1.0f) - 'discnt) as 'revenue) val result = items .groupBy('oId, 'oDate, 'shipPrio) .select('oId, 'revenue.sum, 'oDate, 'shipPrio) 13
- 14. WHAT IS HAPPENING INSIDE? Processing Engine 14
- 15. System Architecture Coordination Processing Optimization Client (pre-flight) Master Workers ... Flink Program ... 15
- 16. Lots of cool technology inside Flink • Batch and Streaming in one system • Memory-safe execution • Native data flow iterations • Cost-based data flow optimizer • Flexible windows on data streams • Type extraction and custom serialization utilities • Static code analysis on user functions • and much more... 16
- 17. STREAM AND BATCH IN ONE SYSTEM Pipelined Data Transfer 17
- 18. Stream and Batch in one System • Most systems do either stream or batch • In the past, Flink focused on batch processing – Flink‘s runtime has always done stream processing – Operators pipeline data forward as soon as it is processed – Some operators are blocking (such as sort) • Pipelining gives better performance for many batch workloads – Avoids materialization of large intermediate results 18
- 19. Pipelined Data Transfer Large Input Small Input Small Input Resultjoin Large Input map Interm. DataSet Build HT Result Program Pipelined Execution Pipeline 1 Pipeline 2 joinProbe HT map No intermediate materialization! 19
- 20. Flink Data Stream Processing • Pipelining enables Flink‘s runtime to process streams – API to define stream programs – Operators for stream processing • Stream API and operators are recent contributions – Evolving very quickly under heavy development – Integration with batch mode 20
- 21. MEMORY SAFE EXECUTION Memory Management and Out-of-Core Algorithms 21
- 22. Memory-safe Execution • Challenge of JVM-based data processing systems – OutOfMemoryErrors due to data objects on the heap • Flink runs complex data flows without memory tuning – C++-style memory management – Robust out-of-core algorithms 22
- 23. Managed Memory • Active memory management – Workers allocate 70% of JVM memory as byte arrays – Algorithms serialize data objects into byte arrays – In-memory processing as long as data is small enough – Otherwise partial destaging to disk • Benefits – Safe memory bounds (no OutOfMemoryError) – Scales to very large JVMs – Reduced GC pressure 23
- 24. Going out-of-core Single-core join of 1KB Java objects beyond memory (4 GB) Blue bars are in-memory, orange bars (partially) out-of-core 24
- 25. GRAPH ANALYSIS Native Data Flow Iterations 25
- 26. Native Data Flow Iterations • Many graph and ML algorithms require iterations • Flink features two types of iterations – Bulk iterations – Delta iterations 26 2 1 5 4 3 0.1 0.5 0.2 0.4 0.7 0.3 0.9
- 27. Iterative Data Flows • Flink runs iterations „natively“ as cyclic data flows – No loop unrolling – Operators are scheduled once – Data is fed back through backflow channel – Loop-invariant data is cached • Operator state is preserved across iterations! initial input Iteration head resultreducejoin Iteration tail other datasets 27
- 28. Delta Iterations • Delta iteration computes – Delta update of solution set – Work set for next iteration • Work set drives computations of next iteration – Workload of later iterations significantly reduced – Fast convergence • Applicable to certain problem domains – Graph processing 0 5000000 10000000 15000000 20000000 25000000 30000000 35000000 40000000 45000000 1 6 11 16 21 26 31 36 41 46 5 #ofelementsupdated # of iterations 28
- 29. Iteration Performance PageRank on Twitter Follower Graph 30 Iterations 61 Iterations (Convergence) 29
- 30. WHAT IS COMING NEXT? Roadmap 30
- 31. Just released a new version • 0.9.0-milestone1 preview release just got out! • Features – Table API – Flink ML Library – Gelly Library – Flink on Tez – … 31
- 32. Flink’s Roadmap Mission: Unified stream and batch processing • Exactly-once streaming semantics with flexible state checkpointing • Extending the ML library • Extending graph library • Integration with Apache Zeppelin (incubating) • SQL on top of the Table API • And much more… 32
- 33. tl;dr – What’s worth to remember? • Flink is a general-purpose data analytics system • Unifies streaming and batch processing • Expressive high-level APIs • Robust and fast execution engine 33
- 34. I Flink, do you? ;-) If you find Flink exciting, get involved and start a discussion on Flink‘s ML or stay tuned by subscribing to [email protected] or following @ApacheFlink on Twitter 34
- 35. 35
- 36. BACKUP 36
- 37. Data Flow Optimizer • Database-style optimizations for parallel data flows • Optimizes all batch programs • Optimizations – Task chaining – Join algorithms – Re-use partitioning and sorting for later operations – Caching for iterations 37
- 38. Data Flow Optimizer val orders = … val lineitems = … val filteredOrders = orders .filter(o => dataFormat.parse(l.shipDate).after(date)) .filter(o => o.shipPrio > 2) val lineitemsOfOrders = filteredOrders .join(lineitems) .where(“orderId”).equalTo(“orderId”) .apply((o,l) => new SelectedItem(o.orderDate, l.extdPrice)) val priceSums = lineitemsOfOrders .groupBy(“orderDate”) .sum(“l.extdPrice”); 38
- 39. Data Flow Optimizer DataSource orders.tbl Filter DataSource lineitem.tbl Join Hybrid Hash buildHT probe broadcast forward Combine Reduce sort[0,1] DataSource orders.tbl Filter DataSource lineitem.tbl Join Hybrid Hash buildHT probe hash-part [0] hash-part [0] hash-part [0,1] Reduce sort[0,1] Best plan depends on relative sizes of input filespartial sort[0,1] 39