An Introduction to Distributed Data Streaming
Download as PPTX, PDF1 like733 views
The document provides an overview of distributed data streaming systems, detailing key concepts such as standing queries, data streams, stream operators, and window operations. It discusses the importance of processing continuous data in real-time and introduces various methods for managing and summarizing data, like synopses and partitioning strategies. Additionally, it highlights implementation examples using frameworks like Apache Storm and Flink for handling streaming data applications.
An Introduction to Distributed Data Streaming
- 1. An Introduction to Distributed Data Streaming Elements and Systems Paris Carbone<[email protected]> PhD Candidate KTH Royal Institute of Technology 1
- 2. 2 how to avoid this? Q = + Q
- 3. Motivation 3 Q Q Q = +
- 5. Preliminaries • Data Streaming Paradigm • Incoming data is unbound - continuous arrival • Standing queries are evaluated continuously • Queries operate on the full data stream or on the most recent views of the stream ~ windows 5
- 6. Data Streams Basics • Events/Tuples : elements of computation - respect a schema • Data Streams : unbounded sequences of events • Stream Operators: consume streams and generate new ones. • Events are consumed once - no backtracking! 6 f S1 S2 So S’1 S’2
- 8. Core Abstractions • Windows • Synopses (summary state) • Partitioning 8
- 9. Windows Discussion Why do we need windows? 9
- 10. Windows • We are often interested only in fresh data • f = “average temperature over the last minute every 20 sec” • Range: Most data stream processing systems allow window operations on the most recent history (eg. 1 minute, 1000 tuples) • Slide: The frequency/granularity f is evaluated on a given range 10 #seconds40 80 Average #3 Average #2 0 Average #1 20 60 100 f W: 1min, 20sec
- 11. Window Types 11 #sec 40 80 Average #2 0 Average #1 20 60 100 #sec 40 80 Average #3 Average #2 0 Average #1 20 60 100 #sec 40 80 Average #2 0 Average #1 20 60 100 0 120 0 120 000 Sliding Tumbling Jumping range > slide range = slide range < slide
- 12. Synopses We cannot infinitely store all events seen • Synopsis: A summary of an infinite stream • It is in principle any streaming operator state • Examples: samples, histograms, sketches, state machines… 12 f s a summary of everything seen so far 1. process t, s 2. update s 3. produce t’ t t’ What about window synopses?
- 13. Synopses-Aggregations • Discussion - Rolling Aggregations • Propose a synopsis, s=? when • f= max • f= ArithmeticMean • f= stDev 13
- 14. Synopses-Approximations 14 • Discussion - Approximate Results • Propose a synopsis, s=? when • f= uniform random sample of k records over the whole stream • f= filter distinct records over windows of 1000 records with a 5% error
- 15. Synopses-ML and Graphs 15 • Examples of cool synopses to check out • Sparsifiers/Spanners - approximating graph properties such as shortest paths • Change detectors - detecting concept drift • Incremental decision trees - continuous stream training and classification
- 16. Partitioning • One stream operator is not enough • Data might be too large to process • e.g. very high input rate, too many stream sources • State could possibly not fit in memory 16 f s f s f s parallel instances How do we partition the input streams? f s
- 17. Partitioning • Partitioning defines how we allocate events to each parallel instance. Typical partitioners are: • Broadcast • Shuffle • Key-based 17 f s f s f s f s f s f s P P P by color
- 18. Putting Everything Together 18 Fire Detection Pipeline {area,temp} {area,smoke} {loc,alert!} • operators • synopses • windows • partitioning trigger on detection trigger periodically ?
- 19. Operators 19 A s F s Rolling Arithmetic Mean of Temperatures State Machine-based Fire Alarm {area,temp} {area,avgTemp} {alarm} Src Sensor Data Sources {area,temp} Src {area} Periodic Temperature Updates Smoke Detections trivial… What is the state and its transitions?
- 20. Partitioning • We are only interested in correlating smoke and high temperature within the same area • Events carry area information so we can partition our computation by area 20 Src P key:area
- 21. Windowing • Individual sensor data could be potentially faulty • We need to gather data from all temperature sensors of an area and produce an average • We want fresh average temperatures 21 Src P key:area{area,temp} A s A s w w w = ?
- 22. The Fire Alarm 22 F s T : avgTemp>40 T : avgTemp<40 …TTTSTTSTTTT…. OK HOT SMOKE FIRE T T T S S T T synopsis= 1 state S : Smoke
- 23. Putting Everything Together 23 {area,temp} {area,smoke} Src Src P P A s A s key:area key:area w w F s F s P key:area {area, alert} {area,avg_temp} {area,smoke}
- 24. Systems: The Big Picture 24 Proprietary Open Source Google DataFlow IBM Infosphere Microsoft Azure Flink Storm Samza Spark
- 25. Evolution 25 ’95 Materialised Views ’01 Complex Event Processing ’03 TelegraphCQ ’03 STREAM ’05 Borealis ’15 User-Defined Windows ’12 Policy-Based Windowing ’88 Active DataBases ’88 HiPac ’12 Twitter Storm ’12 IBM System S ’13 Spark Streaming ’14 Apache Flink ’13 Parallel Recovery ’05 Decentralised Stream Queries ’05 High Availability on Streaming concepts systems ’13 Google Millwheel ’13 Discretized Streams ’00 Eddies 02 Aurora ’12 Twitter Storm
- 26. Programming Models 26 Compositional Declarative • Offer basic building blocks for composing custom operators and topologies • Advanced behaviour such as windowing is often missing • Custom Optimisation • Expose a high-level API • Operators are higher order functions on abstract data stream types • Advanced behaviour such as windowing is supported • Self-Optimisation
- 27. Programming Model Types 27 DStream, DataStream, PCollection… • Direct access to the execution graph / topology • Suitable for engineers • Transformations abstract operator details • Suitable for engineers and data analysts
- 28. Standing Queries with Apache Storm 28 • Step1: Implement input (Spouts) and intermediate operators (Bolts) • Step 2: Construct a Topology by combining operators Spout Bolt Bolt Spouts are the topology sources The listen to data feeds Bolts represent all intermediate computation vertices of the topology They do arbitrary data manipulation Each operator can emit/subscribe to Streams (computation results)
- 29. Example: Topology Definition 29 numbers new_numbers numbers new_numbers toFile
- 30. Standing Queries with Apache Flink 30 Flink Runtime Flink Job Graph Builder/Optimiser Flink Client Streamin g Program • Operator fusion • Window Pre-aggregates • Deploy Long Running Tasks • Monitor Execution
- 31. Distributed Stream Execution Paradigms 31 (Hadoop, Spark) (Spark Streaming) 1) Real Streaming (Distributed Data Flow) LONG-LIVED TASK EXECUTION STATE IS KEPT INSIDE TASKS 2) Batched Execution
- 32. Windows in Action 32 • DStreams are already partitioned in time windows • Only time windows supported • Windows decomposed into policies • Policies can be user-defined too range slide
- 34. Partitioning in Action 34 forward() shuffle() broadcast() keyBy() partitionCustom() shuffleGrouping() allGrouping() fieldsGrouping() customGrouping() repartition(num) reduceByKey() updateStateByKey() no fine-grained controlfull control
- 35. Synopses in Action 35 implementing a rolling max per key
- 36. State in Spark? 36 • Streams are partitioned into small batches • There is practically no state kept in workers (stateless) • How do we keep state?? (Spark Streaming) put new states in output RDDdstream.updateStateByKey(…) In S’
- 37. Implementing the alarm in Flink 37
- 39. Unreliable Sources 39 Standing Query add more sensors Q
- 40. recovered! Unreliable Processing 40 Standing Query Q lost smoke events
- 41. Resilient Brokers Main Features • Topic-based partitioned queues • Strongly consistent offset mapping to records 41
- 42. Processing Guarantees • Kafka solves the source consistency problem • How about the rest of the states of the computation ? (e.g. alert operator state) • Each system offers different guarantees 42 Guarantees Technique Storm at least once event dependency tracking Spark exactly once source upstream backup Flink exactly once periodic snapshots
- 44. Research Topics at KTH/SICS • Exactly-Once-Output Guarantees • State management and auto-scaling • Streaming ML pipelines • Streaming Graphs 44