Apache Spark Crash Course
14 likes3,100 views
This document provides an overview of Apache Spark, including its capabilities and components. Spark is an open-source cluster computing framework that allows distributed processing of large datasets across clusters of machines. It supports various data processing workloads including streaming, SQL, machine learning and graph analytics. The document discusses Spark's APIs like DataFrames and its libraries like Spark SQL, Spark Streaming, MLlib and GraphX. It also provides examples of using Spark for tasks like linear regression modeling.
![35 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Scatter 2D Data Visualized
scatterData ç DataFrame
+-----+--------+
|label|features|
+-----+--------+
|-12.0| [-4.9]|
| -6.0| [-4.5]|
| -7.2| [-4.1]|
| -5.0| [-3.2]|
| -2.0| [-3.0]|
| -3.1| [-2.1]|
| -4.0| [-1.5]|
| -2.2| [-1.2]|
| -2.0| [-0.7]|
| 1.0| [-0.5]|
| -0.7| [-0.2]|
...
...
...](https://image.slidesharecdn.com/crashcourse-apachespark-munich-public-170412150529/85/Apache-Spark-Crash-Course-35-320.jpg)
![37 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
Linear Regression (two features)
Coefficients: [0.464, 0.464]
Intercept: 0.0563](https://image.slidesharecdn.com/crashcourse-apachespark-munich-public-170412150529/85/Apache-Spark-Crash-Course-37-320.jpg)
![81 © Hortonworks Inc. 2011 – 2016. All Rights Reserved
FB Sort
à Spark job that reads 60 TB of compressed data
and performs a 90 TB shuffle and sort.
à Largest real-world Spark job to date!
– Databricks’ PetaByte sort was on synthetic data.
à Multiple reliability fixes.
à Spark job that reads 60 TB of compressed data
and performs a 90 TB shuffle and sort.
à Largest real-world Spark job to date!
– Databricks’ PetaByte sort was on synthetic data.
à Multiple reliability fixes.
“Spark could reliably shuffle and sort 90 TB+ intermediate data and run 250,000 tasks in a
single job [...] and it has been running in production for several months.”](https://image.slidesharecdn.com/crashcourse-apachespark-munich-public-170412150529/85/Apache-Spark-Crash-Course-81-320.jpg)
Apache Spark Crash Course
- 1. Robert Hryniewicz Developer Advocate @RobertH8z Apache Spark Crash Course - DataWorks Summit - Munich 2017
- 2. 2 © Hortonworks Inc. 2011 – 2016. All Rights Reserved “Big Data” à Internet of Anything (IoT) – Wind Turbines, Oil Rigs – Beacons, Wearables – Smart Cars à User Generated Content (Social, Web & Mobile) – Twitter, Facebook, Snapchat – Clickstream – Paypal, Venmo 44ZB in 2020
- 3. 3 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Visualizing 44ZB 100 pixels = 1M TB 100 px -> 1M TB assumes 5M pixel resolution screen
- 5. 5 © Hortonworks Inc. 2011 – 2016. All Rights Reserved The “Big Data” Problem à A single machine cannot process or even store all the data! Problem Solution à Distribute data over large clusters Difficulty à How to split work across machines? à Moving data over network is expensive à Must consider data & network locality à How to deal with failures? à How to deal with slow nodes?
- 7. 7 © Hortonworks Inc. 2011 – 2016. All Rights Reserved What Is Apache Spark? Ã Apache open source project originally developed at AMPLab (University of California Berkeley) Ã Unified data processing engine that operates across varied data workloads and platforms
- 8. 8 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Why Apache Spark? Ã Elegant Developer APIs – Single environment for data munging, data wrangling, and Machine Learning (ML) Ã In-memory computation model – Fast! – Effective for iterative computations and ML Ã Machine Learning – Implementation of distributed ML algorithms – Pipeline API (Spark ML)
- 11. 11 © Hortonworks Inc. 2011 – 2016. All Rights Reserved SparkSession à Main entry point for Spark functionality à Allows programming with DataFrame and Dataset APIs – Fewer concepts and constructs a developer has to juggle while interacting with Spark à Represented as spark and auto-initialized in Zeppelin env. What is it?
- 13. 13 © Hortonworks Inc. 2011 – 2016. All Rights Reserved More Flexible Better Storage and Performance///
- 14. 14 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Spark SQL Overview à Spark module for structured data processing (e.g. DB tables, JSON files, CSV) à Three ways to manipulate data: – DataFrames API – SQL queries – Datasets API
- 15. 15 © Hortonworks Inc. 2011 – 2016. All Rights Reserved DataFrames à Distributed collection of data organized into named columns à Conceptually equivalent to a table in relational DB or a data frame in R/Python à API available in Scala, Java, Python, and R Col1 Col2 … … ColN DataFrame Column Row Data is described as a DataFrame with rows, columns, and a schema
- 16. 16 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Sources CSVAvro HIVE Spark SQL Col1 Col2 … … ColN DataFrame Column Row JSON
- 17. 17 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Create a DataFrame val path = "examples/flights.json" val flights = spark.read.json(path) Example
- 19. 19 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Two API Examples: DataFrame and SQL APIs flights.select("Origin", "Dest", "DepDelay”) .filter($"DepDelay" > 15).show(5) Results +------+----+--------+ |Origin|Dest|DepDelay| +------+----+--------+ | IAD| TPA| 19| | IND| BWI| 34| | IND| JAX| 25| | IND| LAS| 67| | IND| MCO| 94| +------+----+--------+ SELECT Origin, Dest, DepDelay FROM flightsView WHERE DepDelay > 15 LIMIT 5 SQL API DataFrame API
- 21. 21 © Hortonworks Inc. 2011 – 2016. All Rights Reserved What is Stream Processing? Batch Processing • Ability to process and analyze data at-rest (stored data) • Request-based, bulk evaluation and short-lived processing • Enabler for Retrospective, Reactive and On-demand Analytics Stream Processing • Ability to ingest, process and analyze data in-motion in real- or near-real-time • Event or micro-batch driven, continuous evaluation and long-lived processing • Enabler for real-time Prospective, Proactive and Predictive Analytics for Next Best Action Stream Processing + Batch Processing = All Data Analytics real-time (now) historical (past)
- 22. 22 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Next Generation Analytics Iterative & Exploratory Data is the structure Traditional Analytics Structured & Repeatable Structure built to store data 22 Modern Data Applications approach to Insights Start with hypothesis Test against selected data Data leads the way Explore all data, identify correlations Analyze after landing… Analyze in motion…
- 23. 23 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Spark Streaming à Extension of Spark Core API à Stream processing of live data streams – Scalable – High-throughput – Fault-tolerant Overview ZeroMQ MQTT No longer supported in Spark 2.x
- 25. 25 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Spark Streaming Discretized Streams (DStreams) à High-level abstraction representing continuous stream of data à Internally represented as a sequence of RDDs à Operation applied on a DStream translates to operations on the underlying RDDs
- 26. 26 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Spark Streaming Example: flatMap operation
- 27. 27 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Spark Streaming à Apply transformations over a sliding window of data, e.g. rolling average Window Operations
- 28. 28 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Challenges in Streaming Data à Consistency à Fault tolerance à Out-of-order data
- 34. 34 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Machine Learning use cases Healthcare • Predict diagnosis • Prioritize screenings • Reduce re-admittance rates Financial services • Fraud Detection/prevention • Predict underwriting risk • New account risk screens Public Sector • Analyze public sentiment • Optimize resource allocation • Law enforcement & security Retail • Product recommendation • Inventory management • Price optimization Telco/mobile • Predict customer churn • Predict equipment failure • Customer behavior analysis Oil & Gas • Predictive maintenance • Seismic data management • Predict well production levels
- 35. 35 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Scatter 2D Data Visualized scatterData ç DataFrame +-----+--------+ |label|features| +-----+--------+ |-12.0| [-4.9]| | -6.0| [-4.5]| | -7.2| [-4.1]| | -5.0| [-3.2]| | -2.0| [-3.0]| | -3.1| [-2.1]| | -4.0| [-1.5]| | -2.2| [-1.2]| | -2.0| [-0.7]| | 1.0| [-0.5]| | -0.7| [-0.2]| ... ... ...
- 36. 36 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Linear Regression Model Training (one feature) Coefficients: 2.81 Intercept: 3.05 y = 2.81x + 3.05 Training Result
- 37. 37 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Linear Regression (two features) Coefficients: [0.464, 0.464] Intercept: 0.0563
- 38. 38 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Spark API for building ML pipelines Feature transform 1 Feature transform 2 Combine features Linear Regression Input DataFrame Input DataFrame Output DataFrame Pipeline Pipeline Model Train Predict Export Model
- 41. 41 © Hortonworks Inc. 2011 – 2016. All Rights Reserved à Page Rank à Topic Modeling (LDA) à Community Detection Source: ampcamp.berkeley.edu
- 43. 43 © Hortonworks Inc. 2011 – 2016. All Rights Reserved What’s Apache Zeppelin? Web-based notebook that enables interactive data analytics. You can make beautiful data-driven, interactive and collaborative documents with SQL, Python, Scala and more
- 49. 49 © Hortonworks Inc. 2011 – 2016. All Rights Reserved What is a Note/Notebook? • A web based GUI for small code snippets • Write code snippets in browser • Zeppelin sends code to backend for execution • Zeppelin gets data back from backend • Zeppelin visualizes data • Zeppelin Note = Set of (Paragraphs/Cells) • Other Features - Sharing/Collaboration/Reports/Import/Export
- 52. 52 © Hortonworks Inc. 2011 – 2016. All Rights Reserved à Zeppelin è Interactive notebook à Spark à YARN è Resource Management à HDFS è Distributed Storage Layer YARN Scala Java Python R APIs Spark Core Engine Spark SQL Spark Streaming MLlib GraphX 1 ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° N HDFS
- 53. 53 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Access patterns enabled by YARN YARN: Data Operating System 1 ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° ° °N HDFS Hadoop Distributed File System Interactive Real-TimeBatch Applications Batch Needs to happen but, no timeframe limitations Interactive Needs to happen at Human time Real-Time Needs to happen at Machine Execution time.
- 54. 54 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Why Apache Spark on YARN? à Resource management à Utilizes existing HDP cluster infrastructure à Scheduling and queues Spark Driver Client Spark Application Master YARN container Spark Executor YARN container Task Task Spark Executor YARN container Task Task Spark Executor YARN container Task Task
- 55. 55 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Why HDFS? Fault Tolerant Distributed Storage • Divide files into big blocks and distribute 3 copies randomly across the cluster • Processing Data Locality • Not Just storage but computation 10110100101 00100111001 11111001010 01110100101 00101100100 10101001100 01010010111 01011101011 11011011010 10110100101 01001010101 01011100100 11010111010 0 Logical File 1 2 3 4 Blocks 1 Cluster 1 1 2 2 2 3 3 34 4 4
- 58. 58 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Hortonworks Cloud Solutions Microsoft AWS Google Managed Azure HDInsight Non-Managed / Marketplace Hortonworks Data Cloud for AWS Cloud IaaS Hortonworks Data Platform (via Ambari and via Cloudbreak)
- 64. 64 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Livy à Livy is the open source REST interface for interacting with Apache Spark from anywhere à Installed as Spark Ambari Service Livy Client HTTP HTTP (RPC) Spark Interactive Session SparkContext Spark Batch Session SparkContext Livy Server
- 65. 65 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Security Across Zeppelin-Livy-Spark Shiro Ispark Group Interpreter SPNego: Kerberos Kerberos Livy APIs Spark on YARN Zeppelin Driver LDAP Livy Server
- 66. 66 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Reasons to Integrate with Livy à Bring Sessions to Apache Zeppelin – Isolation – Session sharing à Enable efficient cluster resource utilization – Default Spark interpreter keeps YARN/Spark job running forever – Livy interpreter recycled after 60 minutes of inactivity (controlled by livy.server.session.timeout ) à To Identity Propagation – Send user identity from Zeppelin > Livy > Spark on YARN
- 67. 67 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Livy Server SparkSession Sharing Session-2 Session-1 SparkSession-1 SparkContext SparkSession-2 SparkContext Client 1 Client 2 Client 3 Session-1 Session-1 Session-2
- 68. 68 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Apache Zeppelin Security: Authentication + SSL Tommy Callahan Zeppelin Spark on YARN LDAP SSL Firewall 1 2 3
- 69. 69 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Apache Zeppelin + Livy End-to-End Security Ispark Group Interpreter SPNego: Kerberos Kerberos/RPC Livy APIs Spark on YARN Zeppelin LDAP Livy Server Job runs as Tommy Callahan Tommy Callahan
- 71. 71 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Modern Data Apps à HDP 2.6 – Batch Processing à HDF 2.1 – Streaming Apps DATA AT REST DATA IN MOTION ACTIONABLE INTELLIGENCE Modern Data Applications
- 75. 75 © Hortonworks Inc. 2011 – 2016. All Rights Reserved High-Level Overview IoT Edge (single node) IoT Edge (single node) IoT Devices IoT Devices NiFi Hub Data Broker Column DB Data Store Live Dashboard Data Center (on prem/cloud) HDFS/S3 HBase/Cassandra
- 77. 77 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Future Tutorials à Deploying Models with Spark Structured Streaming à Predicting Airline Delays with SparkR à Sentiment Analysis with Apache Spark (Gradient Boosting) à Auto Text Classification (Naïve Bayes)
- 78. 78 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Hortonworks Community Connection Read access for everyone, join to participate and be recognized • Full Q&A Platform (like StackOverflow) • Knowledge Base Articles • Code Samples and Repositories
- 79. 79 © Hortonworks Inc. 2011 – 2016. All Rights Reserved Community Engagement Participate now at: community.hortonworks.com© Hortonworks Inc. 2011 – 2015. All Rights Reserved 12,000+ Registered Users 35,000+ Answers 55,000+ Technical Assets One Website!
- 80. 80 © Hortonworks Inc. 2011 – 2016. All Rights Reserved www.futureofdata.io Future of Data Meetups
- 81. 81 © Hortonworks Inc. 2011 – 2016. All Rights Reserved FB Sort à Spark job that reads 60 TB of compressed data and performs a 90 TB shuffle and sort. à Largest real-world Spark job to date! – Databricks’ PetaByte sort was on synthetic data. à Multiple reliability fixes. à Spark job that reads 60 TB of compressed data and performs a 90 TB shuffle and sort. à Largest real-world Spark job to date! – Databricks’ PetaByte sort was on synthetic data. à Multiple reliability fixes. “Spark could reliably shuffle and sort 90 TB+ intermediate data and run 250,000 tasks in a single job [...] and it has been running in production for several months.”
- 83. 83 © Hortonworks Inc. 2011 – 2016. All Rights Reserved What’s new in HDP 2.6 – Spark & Zeppelin à Spark 1.6.3 GA à Spark 2.1 GA à REST API (Livy) GA à Spark Thrift Server doAS GA à SparkSQL – Row/Column Security (GA) à Spark Streaming + Kafka over SSL à Multi Cluster HBase support for SHC à Package support in PySpark & SparkR Spark à Spark 2.x support à Improved Livy integration à No password in clear à JDBC interpreter improvements à Smart Sense integration à Knox proxy Zeppelin UI Zeppelin 0.7.x