Spark Summit EU 2016 Keynote - Simplifying Big Data in Apache Spark 2.0
6 likes2,788 views
Apache Spark 2.0 simplifies big data processing with a unified engine that enhances productivity and performance through improvements in high-level APIs like DataFrames and structured streaming. New features include whole-stage code generation, support for structured streaming, and optimizations that increase performance significantly, allowing for complex operations such as joins and aggregations. The version is designed to integrate seamlessly with existing applications and support diverse workloads, ultimately facilitating both batch and real-time processing.
![New in 2.0
Whole-stage code generation
• Fuse across multiple operators
• Optimized Parquet I/O
Spark 1.6 14M
rows/s
Spark 2.0 125M
rows/s
Parquet
in 1.6
11M
rows/s
Parquet
in 2.0
90M
rows/s
Merging DataFrame & Dataset
• DataFrame = Dataset[Row]](https://image.slidesharecdn.com/sparksummiteu2016keynote-simplifyingbigdatainapachespark2-161026131548/85/Spark-Summit-EU-2016-Keynote-Simplifying-Big-Data-in-Apache-Spark-2-0-11-320.jpg)
Spark Summit EU 2016 Keynote - Simplifying Big Data in Apache Spark 2.0
- 1. Matei Zaharia @matei_zaharia Simplifying Big Data in Apache Spark 2.0
- 2. A Great Year for Apache Spark 2015 2016 Meetup Members 2015 2016 Developers Contributing 225K 66K 600 1100 2.0 New Major Version #
- 3. About Spark 2.0 Remains highly compatible with 1.x Builds on key lessonsand simplifies API 2000 patches from 280 contributors
- 4. What’s Hard About Big Data? Complex combination of processing tasks, storage systems & modes • ETL, aggregation,machine learning,streaming,etc Hard to get both productivity and performance
- 5. Apache Spark’s Approach Unified engine • Express entire workflow in one API • Connectexisting libraries& storage High-level APIs with space to optimize • RDDs, DataFrames, ML pipelines SQLStreaming ML Graph …
- 6. New in 2.0 Structured API improvements (DataFrame, Dataset, SQL) Whole-stage code generation Structured Streaming Simpler setup (SparkSession) SQL 2003 support MLlib model persistence MLlib R bindings SparkR user-defined functions …
- 7. Original Spark API Arbitrary Java functions on Java objects + Can organize your app using functions, classesand types – Difficult for the engine to optimize • Inefficientin-memory format • Hard to do cross-operatoroptimizations val lines = sc.textFile(“s3://...”) val points = lines.map(line => new Point(line))
- 8. Structured APIs New APIs for data with a fixed schema (table-like) • Efficientstorage taking advantage ofschema (e.g.columnar) • Operators take expressionsin a special DSL thatSpark can optimize DataFrames (untyped), Datasets (typed), and SQL
- 9. Structured API Example events = sc.read.json(“/logs”) stats = events.join(users) .groupBy(“loc”,“status”) .avg(“duration”) errors = stats.where( stats.status == “ERR”) DataFrame API Optimized Plan Specialized Code SCAN logs SCAN users JOIN AGG FILTER while(logs.hasNext) { e = logs.next if(e.status == “ERR”) { u = users.get(e.uid) key = (u.loc, e.status) sum(key) += e.duration count(key) += 1 } } ...
- 10. Structured API Example events = sc.read.json(“/logs”) stats = events.join(users) .groupBy(“loc”,“status”) .avg(“duration”) errors = stats.where( stats.status == “ERR”) DataFrame API Optimized Plan Specialized Code FILTERED SCAN SCAN users JOIN AGG while(logs.hasNext) { e = logs.next if(e.status == “ERR”) { u = users.get(e.uid) key = (u.loc, e.status) sum(key) += e.duration count(key) += 1 } } ...
- 11. New in 2.0 Whole-stage code generation • Fuse across multiple operators • Optimized Parquet I/O Spark 1.6 14M rows/s Spark 2.0 125M rows/s Parquet in 1.6 11M rows/s Parquet in 2.0 90M rows/s Merging DataFrame & Dataset • DataFrame = Dataset[Row]
- 13. Beyond Batch & Interactive: Higher-Level API for Streaming
- 14. What’s Hard In Using Streaming? Complex semantics • What possible resultscan the programgive? • What happensif a node runs slowly? If one fails? Integration into a complete application • Serve real-time querieson resultof stream • Give consistentresultswith batch jobs
- 15. Structured Streaming High-levelstreaming APIbasedon DataFrames / Datasets • Same semantics& results as batch APIs • Eventtime, windowing,sessions,transactionalI/O Rich integration with complete Apache Spark apps • Memory sink forad-hoc queries • Joinswith static data • Change queriesat runtime Not just streaming, but “continuous applications”
- 16. Structured Streaming API Incrementalizean existing DataFrame/Dataset/SQL query logs = ctx.read.format(“json”).open(“hdfs://logs”) logs.groupBy(“userid”, “hour”).avg(“latency”) .write.format(”parquet”) .save(“s3://...”) Example batch job:
- 17. Structured Streaming API Incrementalizean existing DataFrame/Dataset/SQL query logs = ctx.readStream.format(“json”).load(“hdfs://logs”) logs.groupBy(“userid”, “hour”).avg(“latency”) .writeStream.format(”parquet") .start(“s3://...”) Example as streaming: Results always same as a batch job on a prefixof the data
- 18. Under the Hood Scan Files Aggregate Write to S3 Scan New Files Stateful Aggregate Update S3 Batch Plan Continuous Plan Automatically transformed
- 19. Ad-hoc Queries Input Stream Output Sink Streaming Computation Input Stream Output Sink Continuous Application Static Data Batch Job >_ Pure Streaming System ContinuousApplication consistent with End Goal: Full Continuous Apps
- 20. Development Status 2.0.1: supports ETL workloads from file systems and S3 2.0.2: Kafka input source,monitoring metrics 2.1.0: eventtime aggregation workloads & watermarks
- 21. Greg Owen Demo