Introduction to Structured Streaming
3 likes1,652 views
Spark Streaming 2.0 introduces Structured Streaming which addresses some areas for improvement in Spark Streaming 1.X. Structured Streaming builds streaming queries on the Spark SQL engine, providing implicit benefits like extending the primary batch API to streaming and gaining an optimizer. It introduces a more seamless API between batch and stream processing, supports event time semantics, and provides end-to-end fault tolerance guarantees through checkpointing. Structured Streaming also aims to simplify streaming application development by managing streaming queries and allowing continuous queries to be started, stopped, and modified more gracefully.
Introduction to Structured Streaming
- 1. © 2015 IBM Corporation1 ! Agenda - Spark Streaming 1.X • Features • Areas for Improvement - Spark Streaming 2.0 – Structured Streaming • Addressing the Improvement Areas • API • Fault Tolerance • Event Time • Managing Streaming queries - Structured Streaming Examples https://github.com/agsachin/spark-meetup/tree/master/sparkStructuredStreaming - Summary thoughts
- 2. © 2015 IBM Corporation2 Spark Streaming 1. X ! Features of Spark Streaming - High Level API (stateful, joins, aggregates, windows etc.) • Overlap with RDD API (batch) - Fault – Tolerant (exactly once semantics achievable) - Back Pressure - Deep Integration with Spark Ecosystem (MLlib, SQL, GraphX etc.) ! Apache Hadoop Day 2015
- 3. © 2015 IBM Corporation3 Spark Streaming 1. X – Areas of improvement ! Fault-tolerance For end-2-end exactly once guarantees, user needs to do all the heavy lifting in the Sink Can that be handled in a very simple way for the end-user ? Apache Hadoop Day 2015
- 4. © 2015 IBM Corporation4 Fault-Tolerant Semantics Exactly Once, If Outputs are Idempotent or transac6onal Exactly Once, as long as received data is not lost Exactly Once needs re-‐playable sources (e.g. Ka?a Direct) Source Receiver Transforming Outputting Sink
- 5. © 2015 IBM Corporation5 Spark Streaming 1. X – Areas of improvement ! Fault-tolerance - For end-2-end exactly once guarantees, user needs to do all the heavy lifting in the Sink ! API - Request for more seamless API between Batch & Stream - Reduce complexities of streaming app * ! No Event Time support - Hard to support when processing time/batch time exposed in externals ! Streaming Query Management ! Micro-batch ! Apache Hadoop Day 2015
- 6. © 2015 IBM Corporation6 Spark Streaming 2.0 API ! Built on top of Spark SQL Engine ! Implicit Benefits - Extend the primary Batch API even to Streaming - Gain an Optimizer and all other enhancements done in SparkSQL. ! Challenge - Remove/Keep streaming complexities to minimum !
- 7. © 2015 IBM Corporation7 Lets Dive in
- 8. © 2015 IBM Corporation8 SQL Batch vs SQL Streaming- Conceptually
- 9. © 2015 IBM Corporation9 Batch vs Streaming - Programmatically
- 10. © 2015 IBM Corporation10 Output Modes - Sink ! Defined as what gets written from the Result table to external storage (Sink) ! Output modes - Complete – Entire updated Result table is written to external storage. - Append – Only new rows added in the Result table since last incremental query execution is written to external storage. - Update - Only the rows updated in the Result table since last incremental query execution is written to external storage. Upto implementation of Storage connector to decide how to write. * Aggregate queries only support complete mode and non-aggregate queries append mode
- 11. © 2015 IBM Corporation11 Supported Sinks & Modes in 2.0 *DEBUG ONLY *DEBUG ONLY
- 12. © 2015 IBM Corporation12 Windowing in Structured Streaming
- 13. © 2015 IBM Corporation13 Window operations ! Continuous time based aggregations are most common in Streaming applications. - Sliding window & Tumbling window E.g. Top x hashtags on Twitter in last half hour, every 5 minutes ! New function that treats windowing as a regular aggregation ! Used in a Group By clause Can be used in Batch as well
- 14. © 2015 IBM Corporation14 Event Time Windows ! Event-Time is time embedded within the data itself It is not the time Spark received the data ! What about processing time windows if you want them
- 15. © 2015 IBM Corporation15 Handling Late Arrival in Event-Time ! Since the ‘Result’ table is updated by Spark, the late data is put in its correct window group ! Use a normal filter in the SQL ? ! Watermarks
- 16. © 2015 IBM Corporation16 Fault Tolerance ! Why Care? ! Different guarantees for Data Loss ! Atleast Once ! Exactly Once ! What all can fail? ! Driver ! Executor
- 17. © 2015 IBM Corporation17 Spark 1.x Best Fault tolerance - Kafka Direct API • Simplified Parallelism • Less Storage Need • Exactly Once Semantics. source & processing Benefits of this approach
- 18. © 2015 IBM Corporation18 Fault Tolerance in Structured Streaming Active Driver Checkpoint to HDFS ! Structured Streaming Checkpointing Decided Offsets ranges for a trigger interval is logged to checkpoint Directory *before* any processing is started for that trigger Nth record in log indicates data that is currently being processed N-1 entry in log indicates offsets idempotent written to Sink Log entries are monotonically increasing integers ! On Recovery Restart processing of nth entry in WAL
- 19. © 2015 IBM Corporation19 Fault Tolerance in Structured Streaming ! End-to-End Exactly Once guarantees with - idempotent Sinks (built-in for commonly used sinks e.g. Files / JDBC) - Built-in Sources will *mostly* be only ones that support replay https://issues.apache.org/jira/browse/SPARK-15842
- 20. © 2015 IBM Corporation20 Managing Streaming Queries ! Streaming in 1.x was definetly lacking in - Starting / Stopping individual Streaming Queries - Changing the computation done in a Query. - When a Streaming Query abnormally terminates handle more gracefully than app crash.
- 21. © 2015 IBM Corporation21 Managing Streaming Queries
- 22. © 2015 IBM Corporation22 Managing Streaming Queries
- 23. © 2015 IBM Corporation23 Summary ! Overall has a good set of features - Easier code share between Batch and Streaming (No different type hierarchies) - Window not tied to Batch interval - No Streaming context - Optimizer now available for your queries. ! Getting started - Combining of 3 things (Output Mode & Sink Type & Query type) needs some time to wrap your head around * And not much control over those. - Only get Runtime exceptions when you mess with above ! How does it compare to Apache Beam ?
- 24. © 2015 IBM Corporation24 For Each Sink
- 25. © 2015 IBM Corporation25 Thank YOU