Vancouver AWS Meetup Slides 11-20-2018 Apache Spark with Amazon EMR
- 1. An Introduction to Apache Spark with Amazon EMR Peter Smith, Principal Software Engineer, ACL
- 2. Overview • What is Spark? • A Brief Timeline of Spark • Storing data in RDDs and DataFrames • Distributed Processing of Distributed Data • Loading data: CSV, Parquet, JDBC • Queries: SQL, Scala, PySpark • Setting up Spark with EMR (Demo)
- 3. What is Spark? Apache Spark is a unified analytics engine for large-scale data processing. • Can process Terabytes of data (billions of rows) • Click streams from a web application. • IoT data. • Financial trades. • Computation performed over multiple (potentially 1000s) of compute nodes. • Has held the world record for sorting 100TB of data in 23 minutes (2014)
- 4. Usage Scenarios • Batch Processing – Large amounts of data, read from disk, then processed. • Streaming Data – Data is processed in real-time • Machine Learning – Predicting outcomes based on past experience • Graph Processing – Arbitrary data relationship, not just rows and columns
- 5. Spark versus Database SQL Interpreter / Execution Engine SQL Database SQL Client SQL Select, Insert, Update • Disk files in proprietary format (e.g. B-Trees, WALs) • Users never look directly at data files. • Execution engine has 100% control over file storage. • Often database server is a single machine (lots of CPUs)
- 6. Spark versus Database Spark Worker Spark Worker Spark Worker Spark Worker Spark Driver SQL, Java, Scala, Python, R S3 Bucket • Disk formats and locations are 100% controlled by user. • No Transactional Inserts or Updates! • Compute is spread over multiple servers to improve scale. Separate EC2 Servers Amazon EFS
- 7. A Brief Timeline of Spark 2003 - Google File System 2004 - Map Reduce 2006 2010 2013 2011 Hadoop 2.8.4 Spark 2.3.2 (as of today)
- 8. How is Data Stored? Spark allows data to be read or written from disk in a range of formats: • CSV – Possibly the simplest and most common: Fred,Jones,1998,10,Stan • JSON – Often generated by web applications. { “first_name”: “Fred”, … } • JDBC – If the source data is in a database. • Parquet and ORC – Optimized storage for column-oriented queries. • Others – You’re free to write your own connectors. Data is read or written as complete files – doesn’t support inserts or updates. (unlike a transactional database that completely controls the structure of the data).
- 9. RDDs - How Spark Stores Data • RDD – Resilient Distributed Dataset • Data is stored in RAM, partitioned across multiple servers • Each partition operates in parallel. Instead of using database replication (for resilience), Spark will re-perform the work on a different worker.
- 10. Example: Sort people by age. 1. Divide into partitions of 8 people 2. Within group, sort by age. 3. Shuffle people based on decade of birth. 4. Sort within each group.
- 11. Data Frames – Rows/Columns (> Spark v1.5) • RDD – Rows of Java Objects • DataFrame – Rows of Typed Fields (like a Database table) Id (Integer) First_name (String) Last_name (String) BirthYear (Integer) Shoe size (Float) Dog’s name (String) 1 Fran Brown 1982 10.5 Stan 2 Mary Jones 1976 9.0 Fido 3 Brad Pitt 1963 11.0 Barker 4 Jane Simpson 1988 8.0 Rex … … … … … … • DataFrames allow better type-safety and performance optimization.
- 12. Example: Loading data from a CSV File data.csv: 1,Fran,Brown,1982,10.5,Stan 2,Mary,Jones,1976,9.0,Fido 3,Brad,Pitt,1963,11.0,Barker 4,Jane,Simpson,1988,8.0,Rex 5,James,Thompson,1980,9.5,Bif 6,Paul,Wilson,1967,8.5,Ariel 7,Alice,Carlton,1984,11.5,Hank 8,Mike,Taylor,1981,9.5,Quincy 9,Shona,Smith,1975,9.0,Juneau 10,Phil,Arnold,1978,10.0,Koda
- 13. Example: Loading data from a CSV File $ spark-shell scala> val df = spark.read.csv("data.csv") Notes: • Similar methods exist for JSON, JDBC, Parquet, etc. • You can write your own! • Scala is a general purpose programming language (not like SQL)
- 14. Example: Examing a Data Frame scala> df.show(5) +---+-----+--------+----+----+------+ |_c0| _c1| _c2| _c3| _c4| _c5| +---+-----+--------+----+----+------+ | 1| Fran| Brown|1982|10.5| Stan| | 2| Mary| Jones|1976| 9.0| Fido| | 3| Brad| Pitt|1963|11.0|Barker| | 4| Jane| Simpson|1988| 8.0| Rex| | 5|James|Thompson|1980| 9.5| Bif| +---+-----+--------+----+----+------+
- 15. Example: Defining a Schema scala> df.printSchema root |-- _c0: string (nullable = true) |-- _c1: string (nullable = true) |-- _c2: string (nullable = true) |-- _c3: string (nullable = true) |-- _c4: string (nullable = true) |-- _c5: string (nullable = true)
- 16. Example: Defining a Schema scala> df.printSchema root |-- _c0: string (nullable = true) |-- _c1: string (nullable = true) |-- _c2: string (nullable = true) |-- _c3: string (nullable = true) |-- _c4: string (nullable = true) |-- _c5: string (nullable = true)
- 17. Example: Defining a Schema scala> val mySchema = StructType( Array( StructField("id", LongType), StructField("first_name", StringType), StructField("last_name", StringType), StructField("birth_year", IntegerType), StructField("shoe_size", FloatType), StructField("dog_name", StringType) ) ) scala> val df = spark.read.schema(mySchema).csv("data.csv")
- 18. Example: Defining a Schema scala> df.show(5) +---+----------+---------+----------+---------+--------+ | id|first_name|last_name|birth_year|shoe_size|dog_name| +---+----------+---------+----------+---------+--------+ | 1| Fran| Brown| 1982| 10.5| Stan| | 2| Mary| Jones| 1976| 9.0| Fido| | 3| Brad| Pitt| 1963| 11.0| Barker| | 4| Jane| Simpson| 1988| 8.0| Rex| | 5| James| Thompson| 1980| 9.5| Bif| +---+----------+---------+----------+---------+--------+ scala> df.printSchema root |-- id: long (nullable = true) |-- first_name: string (nullable = true) |-- last_name: string (nullable = true) |-- birth_year: integer (nullable = true) |-- shoe_size: float (nullable = true) |-- dog_name: string (nullable = true)
- 19. Example: Counting Records scala> df.count() res21: Long = 10 Imagine 10 Billion rows over 1000 servers?
- 20. Example: Selecting Columns scala> val df_dog = df.select( col("first_name"), col("dog_name”)) scala> df_dog.show(5) +----------+--------+ |first_name|dog_name| +----------+--------+ | Fran| Stan| | Mary| Fido| | Brad| Barker| | Jane| Rex| | James| Bif| +----------+--------+
- 21. Example: Aggregations scala> df.agg( min(col("birth_year")), avg(col("birth_year")) ).show +---------------+---------------+ |min(birth_year)|avg(birth_year)| +---------------+---------------+ | 1963| 1977.4| +---------------+---------------+
- 22. Example: Filtering scala> df.where("birth_year > 1980").show +---+----------+---------+----------+---------+--------+ | id|first_name|last_name|birth_year|shoe_size|dog_name| +---+----------+---------+----------+---------+--------+ | 1| Fran| Brown| 1982| 10.5| Stan| | 4| Jane| Simpson| 1988| 8.0| Rex| | 7| Alice| Carlton| 1984| 11.5| Hank| | 8| Mike| Taylor| 1981| 9.5| Quincy| +---+----------+---------+----------+---------+--------+
- 23. Example: Grouping scala> df.groupBy( (floor(col("birth_year") / 10) * 10) as "Decade" ).count.show +------+-----+ |Decade|count| +------+-----+ | 1960| 2| | 1970| 3| | 1980| 5| +------+-----+
- 24. Example: More Advanced scala> df.select( col("first_name"), col("dog_name"), levenshtein( col("first_name"), col("dog_name") ) as "Diff" ).show(5) +----------+--------+----+ |first_name|dog_name|Diff| +----------+--------+----+ | Fran| Stan| 2| | Mary| Fido| 4| | Brad| Barker| 4| | Jane| Rex| 4| | James| Bif| 5| +----------+--------+----+ Shorter version: df.select( 'first_name, 'dog_name, levenshtein( 'first_name, 'dog_name) as "Diff” ).show(5)
- 25. Queries: User Defined Functions def taxRateFunc(year: Int) = { if (year >= 1984) 0.20 else 0.05 } val taxRate = udf(taxRateFunc _) df.select('birth_year, taxRate('birth_year)).show(5) +----------+---------------+ |birth_year|UDF(birth_year)| +----------+---------------+ | 1982| 0.05| | 1976| 0.05| | 1963| 0.05| | 1988| 0.20| | 1980| 0.05| +----------+---------------+ UDAFs - Check out http://build.acl.com Computing Average Dates in Spark!
- 26. Why is Spark better than a Database? It looks a lot like SQL, but: • Can read/write data in arbitrary formats. • Can be extended with general purpose program code. • Can be split across 1000s of compute nodes. • Can do ML, Streaming, Graph queries. • Can use cheap storage (such as S3) But yeah, if you’re happy with your database, that’s OK too.
- 27. Queries: PySpark Very similar API, but written in Python: $ pyspark >>>> spark.read.csv(”data.csv").show(5) +---+-----+--------+----+----+------+ |_c0| _c1| _c2| _c3| _c4| _c5| +---+-----+--------+----+----+------+ | 1| Fran| Brown|1982|10.5| Stan| | 2| Mary| Jones|1976| 9.0| Fido| | 3| Brad| Pitt|1963|11.0|Barker| | 4| Jane| Simpson|1988| 8.0| Rex| | 5|James|Thompson|1980| 9.5| Bif| +---+-----+--------+----+----+------+
- 28. Demo Time…