SlideShare a Scribd company logo
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING
5 Best Practices in DevOps Culture
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING
What to expect?
Why Spark SQL
1
Use Case
5
Hands-On
Examples
4
Spark SQL Features
3
Spark SQL Libraries
2
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING
Why Spark SQL?
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING
Why Do We Need Spark SQL?
Spark SQL was built to overcome the limitations of Apache Hive
running on top of Spark.
Limitations of Apache
Hive
Hive uses MapReduce which lags in performance with
medium and small sized datasets ( <200 GB)
No resume capability
Hive cannot drop encrypted databases
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING
Spark SQL Advantages Over Hive
Spark SQL uses the metastore services of Hive to query the data stored and managed
by Hive.
Advantages
How?
Faster execution 600 secs
50 secs
1
No migration hurdles
2
Real time querying
3
Batch
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING
Spark SQL
Success Story
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING
Spark SQL Success Story
Twitter Sentiment Analysis
With Spark SQL
Trending Topics can be
used to create
campaigns and attract
larger audience
Sentiment helps in
crisis management,
service adjusting and
target marketing
NYSE: Real Time Analysis of
Stock Market Data
Banking: Credit Card Fraud
Detection
Genomic Sequencing
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING
Spark SQL Features
SQL Integration With Spark
Uniform Data Access
Seamless Support
Transformations
Performance
Standard Connectivity
User Defined Functions
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING
Spark SQL Features
Spark SQL is used for the structured/semi structured data analysis in Spark.
Spark SQL integrates relational processing with Spark’s functional programming.1
2
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING
Spark SQL Features
SQL queries can be converted into RDDs for transformations
Support for various data formats3
4
RDD 1 RDD 2
Shuffle
transform
Drop split
point
Invoking RDD 2 computes all partitions of RDD 1
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING
5 Performance And Scalability
Spark SQL Overview
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING
Spark SQL Features
Standard JDBC/ODBC Connectivity6
7
User Defined Functions lets users define new
Column-based functions to extend the Spark
vocabulary
User
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING
UDF Example
Creating a UDF ā€˜toUpperCase’ to
convert a string to upper case
Registering our UDF in the list of
functions
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING
Spark SQL Architecture
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING
Spark SQL Architecture
Architecture Of Spark SQL
DataFrame DSL Spark SQL & HQL
DataFrame API
Data Source API
CSV JDBCJSON
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING
Spark SQL Libraries
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING
Spark SQL Libraries
Spark SQL has the following libraries:
1 Data Source API
DataFrame API
Interpreter & Optimizer
SQL Service
2
3
4
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING
Data Source API
DataFrame API
Interpreter & Optimizer
SQL Service
Data Source API
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING
Data Source API
Data Source API is used to read and store structured and semi-
structured data into Spark SQL
Features:
 Structured/ Semi-structured data
 Multiple formats
 3rd party integration
Data Source API
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING
Data Source API
DataFrame API
Interpreter & Optimizer
SQL Service
DataFrame API
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING
DataFrame API
DataFrame API converts the data that is read through Data Source API into
tabular columns to help perform SQL operations
Features:
 Distributed collection of data organized into named columns
 Equivalent to a relational table in SQL
 Lazily evaluated
DataFrame API
Named
Columns
Data Source
API
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING
Data Source API
DataFrame API
Interpreter & Optimizer
SQL Service
SQL Interpreter & Optimizer
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING
SQL Interpreter & Optimizer
SQL Interpreter & Optimizer handles the functional programming part of Spark SQL.
It transforms the DataFrames RDDs to get the required results in the required
formats.
Features:
 Functional programming
 Transforming trees
 Faster than RDDs
 Processes all size data
e.g. Catalyst: A modular library for distinct optimization
Interpreter &
Optimizer
Resilient
Distributed
Dataset
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING
Data Source API
DataFrame API
Interpreter & Optimizer
SQL Service
SQL Service
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING
SQL Service
Spark SQL
Service
Interpreter
& Optimizer
Resilient
Distributed
Dataset
 SQL Service is the entry point for working along structured data in Spark
 SQL is used to fetch the result from the interpreted & optimized data
We have thus used all the four libraries in sequence. This completes a Spark SQL
process
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING
Starting Up Spark Shell
Creating Dataset
Adding Schema To RDD
JSON Dataset
Hive Tables
Querying Using Spark SQL
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING
Starting Up Spark Shell
Creating Dataset
Adding Schema To RDD
JSON Dataset
Hive Tables
Starting Up Spark Shell
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING
Starting Up Spark Shell - Intialization
//We first import a Spark Session into Apache Spark.
import org.apache.spark.sql.SparkSession
//Creating a Spark Session ā€˜spark’ using the ā€˜builder()’ function.
val spark = SparkSession.builder().appName("Spark SQL basic
example").config("spark.some.config.option", "some-value").getOrCreate()
//Importing the Implicts class into our ā€˜spark’ Session.
import spark.implicits._
//We now create a DataFrame ā€˜df’ and import data from the ’employee.json’ file.
val df = spark.read.json("examples/src/main/resources/employee.json")
//Displaying the DataFrame ā€˜df’. The result is a table of ages and names from our ’employee.json’ file.
df.show()
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING
Starting Up Spark Shell – Spark Session
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING
Starting Up Spark Shell
Creating Dataset
Adding Schema To RDD
JSON Dataset
Hive Tables
Creating Datasets
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING
Creating Dataset - Case Class & Dataset
After understanding DataFrames, let us now move on to Dataset API.
The below code creates a Dataset class in SparkSQL.
//Creating a class ā€˜Employee’ to store name and age of an employee.
case class Employee(name: String, age: Long)
//Assigning a Dataset ā€˜caseClassDS’ to store the record of Andrew.
val caseClassDS = Seq(Employee("Andrew", 55)).toDS()
//Displaying the Dataset ā€˜caseClassDS’.
caseClassDS.show()
//Creating a primitive Dataset to demonstrate mapping of DataFrames into Datasets.
val primitiveDS = Seq(1, 2, 3).toDS()
//Assigning the above sequence into an array.
primitiveDS.map(_ + 1).collect()
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING
Creating Dataset - Case Class & Dataset
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING
Creating Dataset – Reading File
//Setting the path to our JSON file ’employee.json’.
val path = "examples/src/main/resources/employee.json"
//Creating a Dataset and from the file.
val employeeDS = spark.read.json(path).as[Employee]
//Displaying the contents of ’employeeDS’ Dataset.
employeeDS.show()
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING
Creating Dataset – Reading File
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING
Starting Up Spark Shell
Creating Dataset
Adding Schema To RDD
JSON Dataset
Hive Tables
Adding Schema To RDDs
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING
Adding Schema To RDDs – Initialization
//Importing Expression Encoder for RDDs, Encoder library and Implicts class into the shell.
import org.apache.spark.sql.catalyst.encoders.ExpressionEncoder
import org.apache.spark.sql.Encoder
import spark.implicits._
//Creating an ’employeeDF’ DataFrame from ’employee.txt’ and mapping the columns based on delimiter comma ā€˜,’ into a temporary
view ’employee’.
val employeeDF =
spark.sparkContext.textFile("examples/src/main/resources/employee.txt").map(_.split(",")).ma
p(attributes => Employee(attributes(0), attributes(1).trim.toInt)).toDF()
//Creating the temporary view ’employee’.
employeeDF.createOrReplaceTempView("employee")
//Defining a DataFrame ā€˜youngstersDF’ which will contain all the employees between the ages of 18 and 30.
val youngstersDF = spark.sql("SELECT name, age FROM employee WHERE age BETWEEN 18 AND 30")
//Mapping the names from the RDD into ā€˜youngstersDF’ to display the names of youngsters.
youngstersDF.map(youngster => "Name: " + youngster(0)).show()
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING
Adding Schema To RDDs – Initialization
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING
Adding Schema To RDDs - Transformation
//Converting the mapped names into string for transformations.
youngstersDF.map(youngster => "Name: " +
youngster.getAs[String]("name")).show()
//Using the mapEncoder from Implicits class to map the names to the ages.
implicit val mapEncoder =
org.apache.spark.sql.Encoders.kryo[Map[String, Any]]
//Mapping the names to the ages of our ā€˜youngstersDF’ DataFrame. The result is an array with names
mapped to their respective ages.
youngstersDF.map(youngster =>
youngster.getValuesMap[Any](List("name", "age"))).collect()
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING
Adding Schema To RDDs - Transformation
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING
Adding Schema – Reading File & Adding Schema
//Importing the ā€˜types’ class into the Spark Shell.
import org.apache.spark.sql.types._
//Importing ā€˜Row’ class into the Spark Shell. Row is used in mapping RDD Schema.
import org.apache.spark.sql.Row
//Creating a RDD ’employeeRDD’ from the text file ’employee.txt’.
val employeeRDD = spark.sparkContext.textFile("examples/src/main/resources/employee.txt")
//Defining the schema as ā€œname ageā€. This is used to map the columns of the RDD.
val schemaString = "name age"
//Defining ā€˜fields’ RDD which will be the output after mapping the ’employeeRDD’ to the schema ā€˜schemaString’.
val fields = schemaString.split(" ").map(fieldName => StructField(fieldName, StringType,
nullable = true))
//Obtaining the type of ā€˜fields’ RDD into ā€˜schema’.
val schema = StructType(fields)
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING
Adding Schema – Reading File & Adding Schema
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING
Adding Schema – Transformation Result
//We now create a RDD called ā€˜rowRDD’ and transform the ’employeeRDD’ using the ā€˜map’ function into ā€˜rowRDD’.
val rowRDD = employeeRDD.map(_.split(",")).map(attributes => Row(attributes(0),
attributes(1).trim))
//We define a DataFrame ’employeeDF’ and store the RDD schema into it.
val employeeDF = spark.createDataFrame(rowRDD, schema)
//Creating a temporary view of ’employeeDF’ into ’employee’.
employeeDF.createOrReplaceTempView("employee")
//Performing the SQL operation on ’employee’ to display the contents of employee.
val results = spark.sql("SELECT name FROM employee")
//Displaying the names of the previous operation from the ’employee’ view.
results.map(attributes => "Name: " + attributes(0)).show()
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING
Adding Schema – Transformation Result
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING
Starting Up Spark Shell
Creating Dataset
Adding Schema To RDD
JSON Dataset
Hive Tables
JSON Dataset
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING
JSON Data – Loading File
//Importing Implicits class into the shell.
import spark.implicits._
//Creating an ’employeeDF’ DataFrame from our ’employee.json’ file.
val employeeDF =
spark.read.json("examples/src/main/resources/employee.json")
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING
JSON Data – Loading File
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING
JSON Data – Parquet File
//Creating a ā€˜parquetFile’ temporary view of our DataFrame.
employeeDF.write.parquet("employee.parquet")
val parquetFileDF = spark.read.parquet("employee.parquet")
parquetFileDF.createOrReplaceTempView("parquetFile")
//Selecting the names of people between the ages of 18 and 30 from our Parquet file.
val namesDF = spark.sql("SELECT name FROM parquetFile WHERE
age BETWEEN 18 AND 30")
//Displaying the result of the Spark SQL operation.
namesDF.map(attributes => "Name: " + attributes(0)).show()
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING
JSON Data – Parquet File
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING
JSON Dataset – Creating DataFrame
//Setting to path to our ’employee.json’ file.
val path = "examples/src/main/resources/employee.json"
//Creating a DataFrame ’employeeDF’ from our JSON file.
val employeeDF = spark.read.json(path)
//Printing the schema of ’employeeDF’.
employeeDF.printSchema()
//Creating a temporary view of the DataFrame into ’employee’.
employeeDF.createOrReplaceTempView("employee")
//Defining a DataFrame ā€˜youngsterNamesDF’ which stores the names of all the employees between the ages of
18 and 30 present in ’employee’.
val youngsterNamesDF = spark.sql("SELECT name FROM employee WHERE age
BETWEEN 18 AND 30")
//Displaying the contents of our DataFrame.
youngsterNamesDF.show()
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING
JSON Dataset – Creating DataFrame
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING
JSON Dataset – RDD Operation
//Creating a RDD ā€˜otherEmployeeRDD’ which will store the content of employee
George from New Delhi, Delhi.
val otherEmployeeRDD =
spark.sparkContext.makeRDD("""{"name":"George","address":{"
city":"New Delhi","state":"Delhi"}}""" :: Nil)
//Assigning the contents of ā€˜otherEmployeeRDD’ into ā€˜otherEmployee’.
val otherEmployee = spark.read.json(otherEmployeeRDD)
//Displaying the contents of ā€˜otherEmployee’.
otherEmployee.show()
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING
JSON Dataset – RDD Operation
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING
Starting Up Spark Shell
Creating Dataset
Adding Schema To RDD
JSON Dataset
Hive Tables
Hive Tables
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING
Hive Tables – Case Class & Spark Session
//Importing ā€˜Row’ class and Spark Session into the Spark Shell.
import org.apache.spark.sql.Row
import org.apache.spark.sql.SparkSession
//Creating a class ā€˜Record’ with attributes Int and String.
case class Record(key: Int, value: String)
//Setting the location of ā€˜warehouseLocation’ to Spark warehouse.
val warehouseLocation = "spark-warehouse"
//We now build a Spark Session ā€˜spark’ to demonstrate Hive example in Spark SQL.
val spark = SparkSession.builder().appName("Spark Hive
Example").config("spark.sql.warehouse.dir",
warehouseLocation).enableHiveSupport().getOrCreate()
//Importing Implicits class and SQL library into the shell.
import spark.implicits._
import spark.sql
//Creating a table ā€˜src’ with columns to store key and value.
sql("CREATE TABLE IF NOT EXISTS src (key INT, value STRING)")
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING
Hive Tables – Case Class & Spark Session
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING
Hive Tables – SQL Operation
//We now load the data from the examples present in Spark directory into our table ā€˜src’.
sql("LOAD DATA LOCAL INPATH 'examples/src/main/resources/kv1.txt' INTO
TABLE src")
//The contents of ā€˜src’ is displayed below.
sql("SELECT * FROM src").show()
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING
Hive Tables – SQL Operation
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING
Hive Tables – SQL & DataFrame Transformation
//We perform the ā€˜count’ operation to select the number of keys in ā€˜src’ table.
sql("SELECT COUNT(*) FROM src").show()
//We now select all the records with ā€˜key’ value less than 10 and store it in the ā€˜sqlDF’ DataFrame.
val sqlDF = sql("SELECT key, value FROM src WHERE key < 10 ORDER BY key")
//Creating a Dataset ā€˜stringDS’ from ā€˜sqlDF’.
val stringsDS = sqlDF.map {case Row(key: Int, value: String) => s"Key: $key,
Value: $value"}
//Displaying the contents of ā€˜stringDS’ Dataset.
stringsDS.show()
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING
Hive Tables – SQL & DataFrame Transformation
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING
Hive Tables - Result
//We create a DataFrame ā€˜recordsDF’ and store all the records with key values 1 to 100.
val recordsDF = spark.createDataFrame((1 to 100).map(i => Record(i, s"val_$i")))
//Create a temporary view ā€˜records’ of ā€˜recordsDF’ DataFrame.
recordsDF.createOrReplaceTempView("records")
//Displaying the contents of the join of tables ā€˜records’ and ā€˜src’ with ā€˜key’ as the primary key.
sql("SELECT * FROM records r JOIN src s ON r.key = s.key").show()
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING
Hive Tables - Result
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING
Use Case: Stock Market Analysis
With Spark SQL
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING
Use Case: Problem Statement
Computations to be done:
 Compute the average closing price
 List the companies with highest closing prices
 Compute average closing price per month
 List the number of big price rises and falls
 Compute Statistical correlation
We will use Spark SQL to retrieve trends in the stock market data and thus establish a
financial strategy to avoid risky investment
Stock Market trading generates huge real time data. Analysis of this data is the key to
winning over losing.
This real time data is often present in multiple formats. We need to compute the
analysis with ease.
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING
Use Case: Requirements
Process huge data
Process data in real-time
Easy to use and not very complex
Requirements:
Handle input from multiple sources
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING
Use Case: Why Spark SQL?
Process huge data
Process data in real-time
Easy to use and not very complex
Requirements:
Handle input from multiple sources
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING
Use Case: Stock Market Analysis
We will use stock data from yahoo finance for the following stocks:
AAON Inc., AAON
ABAXIS Inc., ABAX
Fastenal Company, FAST
F5 Networks, FFIV
Gilead Sciences, GILD
Microsoft Corporation, MSFT
O'Reilly Automotive, ORLY
PACCAR Inc., PCAR
A. Schulman, SHLM
Wynn Resorts Limited, WYNN
Our Dataset has data from 10 companies trading in NASDAQ
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING
Use Case: Dataset
The Microsoft stocks MSFT.csv file has the following format :
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING
Implementing Stock Analysis
Using Spark SQL
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING
Use Case: Flow Diagram
Huge amount of real
time stock data
1
DataFrame API for
Relational
Processing
2
RDD for Functional
Programming
3
Calculate Company
with Highest Closing
Price / Year
Calculate Average
Closing Price / Year
Calculate Statistical
Correlation between
Companies
Calculate Dates with
Deviation in Stock
Price
5
Spark SQL
Query
Spark SQL
Query
4
4
Query 3 Query 4
Query 1
Query 2
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING
Spark SQL
Query
Use Case: Flow Example
Calculate Average
Closing Price / Year
4
Real Time Stock
Market Data
1
AAON Company
DataFrame
2
JoinClose RDD
3
Result Table
5
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING
Use Case: Starting Spark Shell
Initialization of Spark SQL in
Spark Shell
Starting a Spark Session
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING
Use Case: Creating Case Class
1. Creating Case Class
2. Defining parseStock schema
3. Defining parseRDD
4. Reading AAON.csv into
stocksAAONDF DataFrame
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING
Use Case: Display DataFrame
Displaying DataFrame stocksAAONDF
Similarly we create DataFrames for
every other company
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING
Use Case: Average Monthly Closing
Display the Average of Adjacent
Closing Price for AAON for every
month
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING
Use Case: Steep Change In Graph
When did the closing price for
Microsoft go up or down by
more than 2 dollars in a day?
1. Create ā€˜result’ to select days
when the difference was
greater than 2
2. Displaying the result
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING
Use Case: Join AAON, ABAX & FAST Stocks
We now join AAON, ABAX &
FAST stocks in order to compare
closing prices
1. Create a Union of AAON,
ABAX & FAST stocks as
joinclose
2. Display joinclose
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING
Use Case: Storing joinclose As Parquet
1. Store joinclose as a Parquet
file joinstock.parquet
2. We can then create a
DataFrame to work on the
table
3. Display the DataFrame ā€˜df’.
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING
Use Case: Average Closing Per Year
1. Create newTables containing
the Average Closing Prices of
AAON, ABAX and FAST per
year
2. Display ā€˜newTables’
3. Register ā€˜newTable’ as a
temporary table
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING
Use Case: Transformation
Transformation of ā€˜newTables’ with
Year and corresponding 3
companies’ data into CompanyAll
table
1. Create transformed table
ā€˜CompanyAll’
2. Display ā€˜CompanyAll’
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING
Use Case: Best Of Average Closing
1. Create ā€˜BestCompany’
containing the Best Average
Closing Prices of AAON,
ABAX and FAST per year
2. Display ā€˜BestCompany’
3. Register ā€˜BestCompany’ as a
temporary table
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING
Use Case: Best Performing Company Per Year
Here, we find the company with
the best Closing Price Average per
year
1. Create ā€˜FinalTable’ from the
join of BestCompanyYear and
CompanyAll
2. Displaying FinalTable
3. Register ā€˜FinalTable’
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING
Use Case: Correlation
We use Statistics library to find the
correlation between AAON and ABAX
companies closing prices.
Correlation, in the finance and
investment industries, is a statistic that
measures the degree to which two
securities move in relation to each
other.
The closer the correlation is to 1, the
graph of the stocks follow a similar
trend.
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING
Conclusion
Congrats!
We have hence demonstrated the power of Spark SQL in Real Time Data Analytics for
Stock Market.
The hands-on examples will give you the required confidence to work on any future
projects you encounter in Spark SQL.
www.edureka.co/apache-spark-scala-trainingEDUREKA SPARK CERTIFICATION TRAINING
Thank You …
Questions/Queries/Feedback

More Related Content

What's hot (20)

PPTX
A Deep Dive into Spark SQL's Catalyst Optimizer with Yin Huai
Databricks
Ā 
PDF
Introduction to Spark Streaming
datamantra
Ā 
PDF
Understanding Query Plans and Spark UIs
Databricks
Ā 
PDF
Apache Spark Introduction
sudhakara st
Ā 
PDF
What is Apache Spark | Apache Spark Tutorial For Beginners | Apache Spark Tra...
Edureka!
Ā 
PDF
Pyspark Tutorial | Introduction to Apache Spark with Python | PySpark Trainin...
Edureka!
Ā 
PPTX
Introduction to Apache Spark
Rahul Jain
Ā 
PDF
Apache Spark - Basics of RDD | Big Data Hadoop Spark Tutorial | CloudxLab
CloudxLab
Ā 
PPTX
Apache Spark Components
Girish Khanzode
Ā 
PDF
Introduction to Spark Internals
Pietro Michiardi
Ā 
PDF
Spark streaming , Spark SQL
Yousun Jeong
Ā 
PDF
What Is RDD In Spark? | Edureka
Edureka!
Ā 
PDF
Apache spark
shima jafari
Ā 
PDF
A Tale of Three Apache Spark APIs: RDDs, DataFrames, and Datasets with Jules ...
Databricks
Ā 
PPTX
04 spark-pair rdd-rdd-persistence
Venkat Datla
Ā 
PPT
Apache Spark Introduction and Resilient Distributed Dataset basics and deep dive
Sachin Aggarwal
Ā 
PDF
Spark SQL
Joud Khattab
Ā 
PPTX
Hive Tutorial | Hive Architecture | Hive Tutorial For Beginners | Hive In Had...
Simplilearn
Ā 
PPTX
Spark architecture
GauravBiswas9
Ā 
PPTX
Introduction to ML with Apache Spark MLlib
Taras Matyashovsky
Ā 
A Deep Dive into Spark SQL's Catalyst Optimizer with Yin Huai
Databricks
Ā 
Introduction to Spark Streaming
datamantra
Ā 
Understanding Query Plans and Spark UIs
Databricks
Ā 
Apache Spark Introduction
sudhakara st
Ā 
What is Apache Spark | Apache Spark Tutorial For Beginners | Apache Spark Tra...
Edureka!
Ā 
Pyspark Tutorial | Introduction to Apache Spark with Python | PySpark Trainin...
Edureka!
Ā 
Introduction to Apache Spark
Rahul Jain
Ā 
Apache Spark - Basics of RDD | Big Data Hadoop Spark Tutorial | CloudxLab
CloudxLab
Ā 
Apache Spark Components
Girish Khanzode
Ā 
Introduction to Spark Internals
Pietro Michiardi
Ā 
Spark streaming , Spark SQL
Yousun Jeong
Ā 
What Is RDD In Spark? | Edureka
Edureka!
Ā 
Apache spark
shima jafari
Ā 
A Tale of Three Apache Spark APIs: RDDs, DataFrames, and Datasets with Jules ...
Databricks
Ā 
04 spark-pair rdd-rdd-persistence
Venkat Datla
Ā 
Apache Spark Introduction and Resilient Distributed Dataset basics and deep dive
Sachin Aggarwal
Ā 
Spark SQL
Joud Khattab
Ā 
Hive Tutorial | Hive Architecture | Hive Tutorial For Beginners | Hive In Had...
Simplilearn
Ā 
Spark architecture
GauravBiswas9
Ā 
Introduction to ML with Apache Spark MLlib
Taras Matyashovsky
Ā 

Similar to Spark SQL Tutorial | Spark Tutorial for Beginners | Apache Spark Training | Edureka (20)

PDF
Spark SQL | Apache Spark
Edureka!
Ā 
PDF
Big Data Processing With Spark
Edureka!
Ā 
PDF
Spark For Faster Batch Processing
Edureka!
Ā 
PDF
Big Data Processing with Spark and Scala
Edureka!
Ā 
PDF
Apache Spark Tutorial | Spark Tutorial for Beginners | Apache Spark Training ...
Edureka!
Ā 
PDF
Spark Hadoop Tutorial | Spark Hadoop Example on NBA | Apache Spark Training |...
Edureka!
Ā 
PDF
5 things one must know about spark!
Edureka!
Ā 
PDF
5 Reasons why Spark is in demand!
Edureka!
Ā 
PPTX
Big data Processing with Apache Spark & Scala
Edureka!
Ā 
PPTX
5 things one must know about spark!
Edureka!
Ā 
PDF
Spark Interview Questions and Answers | Apache Spark Interview Questions | Sp...
Edureka!
Ā 
PPTX
Spark sql
Zahra Eskandari
Ā 
PPTX
5 reasons why spark is in demand!
Edureka!
Ā 
PDF
Spark Will Replace Hadoop ! Know Why
Edureka!
Ā 
PDF
Spark is going to replace Apache Hadoop! Know Why?
Edureka!
Ā 
PPTX
Spark for big data analytics
Edureka!
Ā 
PPTX
Apache Spark & Scala
Edureka!
Ā 
PDF
Spark Streaming
Edureka!
Ā 
PPTX
Learning spark ch09 - Spark SQL
phanleson
Ā 
PDF
Jumpstart on Apache Spark 2.2 on Databricks
Databricks
Ā 
Spark SQL | Apache Spark
Edureka!
Ā 
Big Data Processing With Spark
Edureka!
Ā 
Spark For Faster Batch Processing
Edureka!
Ā 
Big Data Processing with Spark and Scala
Edureka!
Ā 
Apache Spark Tutorial | Spark Tutorial for Beginners | Apache Spark Training ...
Edureka!
Ā 
Spark Hadoop Tutorial | Spark Hadoop Example on NBA | Apache Spark Training |...
Edureka!
Ā 
5 things one must know about spark!
Edureka!
Ā 
5 Reasons why Spark is in demand!
Edureka!
Ā 
Big data Processing with Apache Spark & Scala
Edureka!
Ā 
5 things one must know about spark!
Edureka!
Ā 
Spark Interview Questions and Answers | Apache Spark Interview Questions | Sp...
Edureka!
Ā 
Spark sql
Zahra Eskandari
Ā 
5 reasons why spark is in demand!
Edureka!
Ā 
Spark Will Replace Hadoop ! Know Why
Edureka!
Ā 
Spark is going to replace Apache Hadoop! Know Why?
Edureka!
Ā 
Spark for big data analytics
Edureka!
Ā 
Apache Spark & Scala
Edureka!
Ā 
Spark Streaming
Edureka!
Ā 
Learning spark ch09 - Spark SQL
phanleson
Ā 
Jumpstart on Apache Spark 2.2 on Databricks
Databricks
Ā 
Ad

More from Edureka! (20)

PDF
What to learn during the 21 days Lockdown | Edureka
Edureka!
Ā 
PDF
Top 10 Dying Programming Languages in 2020 | Edureka
Edureka!
Ā 
PDF
Top 5 Trending Business Intelligence Tools | Edureka
Edureka!
Ā 
PDF
Tableau Tutorial for Data Science | Edureka
Edureka!
Ā 
PDF
Python Programming Tutorial | Edureka
Edureka!
Ā 
PDF
Top 5 PMP Certifications | Edureka
Edureka!
Ā 
PDF
Top Maven Interview Questions in 2020 | Edureka
Edureka!
Ā 
PDF
Linux Mint Tutorial | Edureka
Edureka!
Ā 
PDF
How to Deploy Java Web App in AWS| Edureka
Edureka!
Ā 
PDF
Importance of Digital Marketing | Edureka
Edureka!
Ā 
PDF
RPA in 2020 | Edureka
Edureka!
Ā 
PDF
Email Notifications in Jenkins | Edureka
Edureka!
Ā 
PDF
EA Algorithm in Machine Learning | Edureka
Edureka!
Ā 
PDF
Cognitive AI Tutorial | Edureka
Edureka!
Ā 
PDF
AWS Cloud Practitioner Tutorial | Edureka
Edureka!
Ā 
PDF
Blue Prism Top Interview Questions | Edureka
Edureka!
Ā 
PDF
Big Data on AWS Tutorial | Edureka
Edureka!
Ā 
PDF
A star algorithm | A* Algorithm in Artificial Intelligence | Edureka
Edureka!
Ā 
PDF
Kubernetes Installation on Ubuntu | Edureka
Edureka!
Ā 
PDF
Introduction to DevOps | Edureka
Edureka!
Ā 
What to learn during the 21 days Lockdown | Edureka
Edureka!
Ā 
Top 10 Dying Programming Languages in 2020 | Edureka
Edureka!
Ā 
Top 5 Trending Business Intelligence Tools | Edureka
Edureka!
Ā 
Tableau Tutorial for Data Science | Edureka
Edureka!
Ā 
Python Programming Tutorial | Edureka
Edureka!
Ā 
Top 5 PMP Certifications | Edureka
Edureka!
Ā 
Top Maven Interview Questions in 2020 | Edureka
Edureka!
Ā 
Linux Mint Tutorial | Edureka
Edureka!
Ā 
How to Deploy Java Web App in AWS| Edureka
Edureka!
Ā 
Importance of Digital Marketing | Edureka
Edureka!
Ā 
RPA in 2020 | Edureka
Edureka!
Ā 
Email Notifications in Jenkins | Edureka
Edureka!
Ā 
EA Algorithm in Machine Learning | Edureka
Edureka!
Ā 
Cognitive AI Tutorial | Edureka
Edureka!
Ā 
AWS Cloud Practitioner Tutorial | Edureka
Edureka!
Ā 
Blue Prism Top Interview Questions | Edureka
Edureka!
Ā 
Big Data on AWS Tutorial | Edureka
Edureka!
Ā 
A star algorithm | A* Algorithm in Artificial Intelligence | Edureka
Edureka!
Ā 
Kubernetes Installation on Ubuntu | Edureka
Edureka!
Ā 
Introduction to DevOps | Edureka
Edureka!
Ā 
Ad

Recently uploaded (20)

PPTX
The Project Compass - GDG on Campus MSIT
dscmsitkol
Ā 
PDF
Future-Proof or Fall Behind? 10 Tech Trends You Can’t Afford to Ignore in 2025
DIGITALCONFEX
Ā 
PDF
NASA A Researcher’s Guide to International Space Station : Physical Sciences ...
Dr. PANKAJ DHUSSA
Ā 
PPTX
From Sci-Fi to Reality: Exploring AI Evolution
Svetlana Meissner
Ā 
PDF
How do you fast track Agentic automation use cases discovery?
DianaGray10
Ā 
PDF
UPDF - AI PDF Editor & Converter Key Features
DealFuel
Ā 
PDF
Peak of Data & AI Encore AI-Enhanced Workflows for the Real World
Safe Software
Ā 
PPTX
COMPARISON OF RASTER ANALYSIS TOOLS OF QGIS AND ARCGIS
Sharanya Sarkar
Ā 
DOCX
Python coding for beginners !! Start now!#
Rajni Bhardwaj Grover
Ā 
PPTX
Agentforce World Tour Toronto '25 - Supercharge MuleSoft Development with Mod...
Alexandra N. Martinez
Ā 
PDF
šŸš€ Let’s Build Our First Slack Workflow! šŸ”§.pdf
SanjeetMishra29
Ā 
PPTX
Digital Circuits, important subject in CS
contactparinay1
Ā 
PDF
CIFDAQ Market Wrap for the week of 4th July 2025
CIFDAQ
Ā 
PDF
[Newgen] NewgenONE Marvin Brochure 1.pdf
darshakparmar
Ā 
PDF
Staying Human in a Machine- Accelerated World
Catalin Jora
Ā 
PPTX
New ThousandEyes Product Innovations: Cisco Live June 2025
ThousandEyes
Ā 
PPTX
Agentforce World Tour Toronto '25 - MCP with MuleSoft
Alexandra N. Martinez
Ā 
PDF
Newgen 2022-Forrester Newgen TEI_13 05 2022-The-Total-Economic-Impact-Newgen-...
darshakparmar
Ā 
PDF
The 2025 InfraRed Report - Redpoint Ventures
Razin Mustafiz
Ā 
PDF
Transcript: Book industry state of the nation 2025 - Tech Forum 2025
BookNet Canada
Ā 
The Project Compass - GDG on Campus MSIT
dscmsitkol
Ā 
Future-Proof or Fall Behind? 10 Tech Trends You Can’t Afford to Ignore in 2025
DIGITALCONFEX
Ā 
NASA A Researcher’s Guide to International Space Station : Physical Sciences ...
Dr. PANKAJ DHUSSA
Ā 
From Sci-Fi to Reality: Exploring AI Evolution
Svetlana Meissner
Ā 
How do you fast track Agentic automation use cases discovery?
DianaGray10
Ā 
UPDF - AI PDF Editor & Converter Key Features
DealFuel
Ā 
Peak of Data & AI Encore AI-Enhanced Workflows for the Real World
Safe Software
Ā 
COMPARISON OF RASTER ANALYSIS TOOLS OF QGIS AND ARCGIS
Sharanya Sarkar
Ā 
Python coding for beginners !! Start now!#
Rajni Bhardwaj Grover
Ā 
Agentforce World Tour Toronto '25 - Supercharge MuleSoft Development with Mod...
Alexandra N. Martinez
Ā 
šŸš€ Let’s Build Our First Slack Workflow! šŸ”§.pdf
SanjeetMishra29
Ā 
Digital Circuits, important subject in CS
contactparinay1
Ā 
CIFDAQ Market Wrap for the week of 4th July 2025
CIFDAQ
Ā 
[Newgen] NewgenONE Marvin Brochure 1.pdf
darshakparmar
Ā 
Staying Human in a Machine- Accelerated World
Catalin Jora
Ā 
New ThousandEyes Product Innovations: Cisco Live June 2025
ThousandEyes
Ā 
Agentforce World Tour Toronto '25 - MCP with MuleSoft
Alexandra N. Martinez
Ā 
Newgen 2022-Forrester Newgen TEI_13 05 2022-The-Total-Economic-Impact-Newgen-...
darshakparmar
Ā 
The 2025 InfraRed Report - Redpoint Ventures
Razin Mustafiz
Ā 
Transcript: Book industry state of the nation 2025 - Tech Forum 2025
BookNet Canada
Ā 

Spark SQL Tutorial | Spark Tutorial for Beginners | Apache Spark Training | Edureka