DIscover Spark and Spark streaming
1 like778 views
This document provides an overview of Spark, including how it differs from MapReduce by loading more data into memory and implementing caching mechanisms. It discusses Spark's ecosystem, features like RDDs, and how it can run on Hadoop clusters. It also summarizes Spark Streaming, describing how it processes data in micro-batches, provides windowing and transformation operations on DStreams, and unifies batch, streaming, and interactive analytics.
DIscover Spark and Spark streaming
- 1. TechLabs by A la découverte de Machine Learning, de Redis et de Spark
- 4. Outline ❏ Data processing today ❏ Spark, hadoop, MapReduce ❏ Spark ecosystem ❏ Spark basics
- 5. Data processing today Data intensive application Definition : “We call an application data-intensive if data is its primary challenge—the quantity of data, the complexity of data, or the speed at which it is changing—as opposed to compute-intensive, where CPU cycles are the bottleneck.” Martin Klepmann
- 6. Data processing today Today apps needs : ❏ Store data (databases) ❏ Caches ❏ search data (search index) ❏ Asynchronously message handling (stream processing) ❏ batch processing
- 8. Spark, hadoop, MapReduce Spark : main differences with Map Reduce ❏ Spark load most of the dataset in memory ❏ Implement cache mechanisms which reduce read from disk ❏ Is much faster than MapReduce : Job scheduling ❏ Does not implement any data distribution technology but can run on top of hadoop clusters (HDFS )
- 9. Spark ecosystem : open source
- 10. Spark ecosystem : features
- 11. Spark ecosystem : deployment
- 12. Spark basics : RDD RDD : Resilient Distributed data ❏ Primary spark abstraction ❏ Fault tolerant collection of elements ❏ Partitioned and Immutable ❏ Two types operations ❏ Lazy Transformation
- 13. Spark basics : An execution flow
- 14. Spark Streaming
- 15. Outline ❏ Why In-stream processing ? ❏ Runtime and Programming Model ❏ Spark Streaming : Overview ❏ Benefits of Discretized Stream Processing ❏ Processing flow ❏ Transform operations ❏ Window operations
- 16. Why In-stream processing ?
- 17. Why In-stream processing ?
- 18. Runtime and Programming Model Native Streaming
- 19. Runtime and Programming Model Micro-batch Streaming
- 20. Spark Streaming : Overview
- 21. Benefits of Discretized Stream Processing Dynamic load balancing
- 22. Benefits of Discretized Stream Processing Fast failure and straggler recovery
- 23. Benefits of Discretized Stream Processing ❏ Unification of batch, streaming and interactive analytics ❏ Advanced analytics like machine learning and interactive SQL ❏ Streaming + SQL and DataFrames ❏ Streaming + MLlib
- 24. Spark Streaming : Processing flow
- 25. Spark Streaming : DStreams Discretized Streams (DStreams) : ❏ The basic spark streaming abstraction ❏ A continuous series of RDDs
- 26. Spark Streaming : Transformations Transform Operations : Any operation applied on a DStream translates to operations on the underlying RDDs
- 27. Spark Streaming : Transformations Window Operations :
- 28. Spark Streaming : Time abstractions Batch interval Sliding interval Window size
- 29. Spark Streaming : Time abstractions Batch interval Window size Sliding interval
- 30. Spark Streaming : Some examples ❏ Wordcount ❏ stateless operation, counting words for every batch ❏ Basic Error count ❏ stateless operation, using a filter : contains(“ERROR”) ❏ Cumulative Error count ❏ Stateful operation, errors from the beginning of the processing ❏ Windowed Errors counts ❏ Stateful operation, errors from the sliding window of time