Javantura v4 - Getting started with Apache Spark - Dinko Srkoč
0 likes1,067 views
This document provides an introduction to Apache Spark and Scala. It discusses that Apache Spark is a general purpose cluster computing system that is faster than Hadoop MapReduce, runs locally and in the cloud. It has high-level APIs for Scala, Python, Java and R. The document outlines Spark's core components including SQL, MLlib, GraphX and streaming. It describes Spark's main data collections of RDDs for unstructured data and DataFrames/Datasets for structured data. Finally, it provides an overview of demonstrations that will be covered including the Spark shell, notebook, streaming and deploying a mini project to Google Cloud Dataproc.
1 of 6
Download to read offline
![Data collections in Spark
Collections: immutable, distributed, partitioned across nodes, operated in parallel
● Resilient Distributed Dataset (RDD)
○ Basic abstraction
○ Low-level API
○ Suitable for unstructured data (media, streams of text)
● Dataset/DataFrame
○ Dataset[T] - typed API, DataFrame (a.k.a. DataSet[Row]) - untyped API
○ High-level expressions: filters/maps, aggregations, averages, SQL queries, columnar access
○ optimizations](https://image.slidesharecdn.com/javanturav4-gettingstartedwithapachesparkdinkosrko-170217082127/85/Javantura-v4-Getting-started-with-Apache-Spark-Dinko-Srkoc-4-320.jpg)
More Related Content
What's hot (20)
PDF
Data Policies for the Kafka-API with WebAssembly | Alexander Gallego, VectorizedHostedbyConfluentViewers also liked (20)
PDF
Javantura v4 - Let me tell you a story why Scrum is not for you - Roko RoićHUJAK - Hrvatska udruga Java korisnika / Croatian Java User AssociationPDF
Javantura v4 - KumuluzEE – Microservices with Java - Matjaž B. Jurič & Tilen ...HUJAK - Hrvatska udruga Java korisnika / Croatian Java User AssociationPDF
Javantura v4 - Support SpringBoot application development lifecycle using Ora...HUJAK - Hrvatska udruga Java korisnika / Croatian Java User AssociationPDF
Javantura v4 - Test-driven documentation with Spring REST Docs - Danijel MitarHUJAK - Hrvatska udruga Java korisnika / Croatian Java User AssociationPDF
Javantura v4 - Angular2 - Ionic2 - from birth to stable versions - Hrvoje Pek...HUJAK - Hrvatska udruga Java korisnika / Croatian Java User AssociationPDF
Javantura v4 - Spring Boot and JavaFX - can they play together - Josip KovačekHUJAK - Hrvatska udruga Java korisnika / Croatian Java User AssociationPDF
Javantura v4 - What’s NOT new in modular Java - Milen DyankovHUJAK - Hrvatska udruga Java korisnika / Croatian Java User AssociationPDF
Javantura v4 - Java and lambdas and streams - are they better than for loops ...HUJAK - Hrvatska udruga Java korisnika / Croatian Java User AssociationPDF
Javantura v4 - Java or Scala – Web development with Playframework 2.5.x - Kre...HUJAK - Hrvatska udruga Java korisnika / Croatian Java User AssociationPDF
Javantura v4 - CroDuke Indy and the Kingdom of Java Skills - Branko Mihaljevi...HUJAK - Hrvatska udruga Java korisnika / Croatian Java User AssociationPDF
Javantura v4 - DMN – supplement your BPMN - Željko ŠmagucHUJAK - Hrvatska udruga Java korisnika / Croatian Java User AssociationPDF
Javantura v4 - (Spring)Boot your application on Red Hat middleware stack - Al...HUJAK - Hrvatska udruga Java korisnika / Croatian Java User AssociationPDF
Javantura v4 - JVM++ The GraalVM - Martin ToshevHUJAK - Hrvatska udruga Java korisnika / Croatian Java User AssociationPDF
Javantura v4 - FreeMarker in Spring web - Marin KalapaćHUJAK - Hrvatska udruga Java korisnika / Croatian Java User AssociationPDF
Javantura v4 - The power of cloud in professional services company - Ivan Krn...HUJAK - Hrvatska udruga Java korisnika / Croatian Java User AssociationPDF
Javantura v4 - Cloud-native Architectures and Java - Matjaž B. JuričHUJAK - Hrvatska udruga Java korisnika / Croatian Java User AssociationPDF
Javantura v4 - True RESTful Java Web Services with JSON API and Katharsis - M...HUJAK - Hrvatska udruga Java korisnika / Croatian Java User AssociationPDF
Javantura v4 - Security architecture of the Java platform - Martin ToshevHUJAK - Hrvatska udruga Java korisnika / Croatian Java User AssociationPDF
Javantura v4 - Keycloak – instant login for your app - Marko ŠtrukeljHUJAK - Hrvatska udruga Java korisnika / Croatian Java User AssociationPDF
Javantura v4 - Android App Development in 2017 - Matej VidakovićHUJAK - Hrvatska udruga Java korisnika / Croatian Java User AssociationJavantura v4 - Let me tell you a story why Scrum is not for you - Roko RoićHUJAK - Hrvatska udruga Java korisnika / Croatian Java User Association
Javantura v4 - KumuluzEE – Microservices with Java - Matjaž B. Jurič & Tilen ...HUJAK - Hrvatska udruga Java korisnika / Croatian Java User Association
Javantura v4 - Support SpringBoot application development lifecycle using Ora...HUJAK - Hrvatska udruga Java korisnika / Croatian Java User Association
Javantura v4 - Test-driven documentation with Spring REST Docs - Danijel MitarHUJAK - Hrvatska udruga Java korisnika / Croatian Java User Association
Javantura v4 - Angular2 - Ionic2 - from birth to stable versions - Hrvoje Pek...HUJAK - Hrvatska udruga Java korisnika / Croatian Java User Association
Javantura v4 - Spring Boot and JavaFX - can they play together - Josip KovačekHUJAK - Hrvatska udruga Java korisnika / Croatian Java User Association
Javantura v4 - What’s NOT new in modular Java - Milen DyankovHUJAK - Hrvatska udruga Java korisnika / Croatian Java User Association
Javantura v4 - Java and lambdas and streams - are they better than for loops ...HUJAK - Hrvatska udruga Java korisnika / Croatian Java User Association
Javantura v4 - Java or Scala – Web development with Playframework 2.5.x - Kre...HUJAK - Hrvatska udruga Java korisnika / Croatian Java User Association
Javantura v4 - CroDuke Indy and the Kingdom of Java Skills - Branko Mihaljevi...HUJAK - Hrvatska udruga Java korisnika / Croatian Java User Association
Javantura v4 - DMN – supplement your BPMN - Željko ŠmagucHUJAK - Hrvatska udruga Java korisnika / Croatian Java User Association
Javantura v4 - (Spring)Boot your application on Red Hat middleware stack - Al...HUJAK - Hrvatska udruga Java korisnika / Croatian Java User Association
Javantura v4 - JVM++ The GraalVM - Martin ToshevHUJAK - Hrvatska udruga Java korisnika / Croatian Java User Association
Javantura v4 - FreeMarker in Spring web - Marin KalapaćHUJAK - Hrvatska udruga Java korisnika / Croatian Java User Association
Javantura v4 - The power of cloud in professional services company - Ivan Krn...HUJAK - Hrvatska udruga Java korisnika / Croatian Java User Association
Javantura v4 - Cloud-native Architectures and Java - Matjaž B. JuričHUJAK - Hrvatska udruga Java korisnika / Croatian Java User Association
Javantura v4 - True RESTful Java Web Services with JSON API and Katharsis - M...HUJAK - Hrvatska udruga Java korisnika / Croatian Java User Association
Javantura v4 - Security architecture of the Java platform - Martin ToshevHUJAK - Hrvatska udruga Java korisnika / Croatian Java User Association
Javantura v4 - Keycloak – instant login for your app - Marko ŠtrukeljHUJAK - Hrvatska udruga Java korisnika / Croatian Java User Association
Javantura v4 - Android App Development in 2017 - Matej VidakovićHUJAK - Hrvatska udruga Java korisnika / Croatian Java User Association
Ad
Similar to Javantura v4 - Getting started with Apache Spark - Dinko Srkoč (20)
PPTX
Apache Spark - Lightning Fast Cluster Computing - Hyderabad Scalability MeetupHyderabad Scalability MeetupApache Spark - Lightning Fast Cluster Computing - Hyderabad Scalability MeetupHyderabad Scalability Meetup
Ad
More from HUJAK - Hrvatska udruga Java korisnika / Croatian Java User Association (20)
PDF
Java cro'21 the best tools for java developers in 2021 - hujakHUJAK - Hrvatska udruga Java korisnika / Croatian Java User AssociationPDF
JavaCro'21 - Java is Here To Stay - HUJAK KeynoteHUJAK - Hrvatska udruga Java korisnika / Croatian Java User AssociationPDF
Javantura v7 - Behaviour Driven Development with Cucumber - Ivan LozićHUJAK - Hrvatska udruga Java korisnika / Croatian Java User AssociationPPTX
Javantura v7 - The State of Java - Today and Tomowwow - HUJAK's Community Key...HUJAK - Hrvatska udruga Java korisnika / Croatian Java User AssociationPPTX
Javantura v7 - Learning to Scale Yourself: The Journey from Coder to Leader -...HUJAK - Hrvatska udruga Java korisnika / Croatian Java User AssociationPDF
JavaCro'19 - The State of Java and Software Development in Croatia - Communit...HUJAK - Hrvatska udruga Java korisnika / Croatian Java User AssociationPDF
Javantura v6 - Java in Croatia and HUJAK - Branko Mihaljević, Aleksander RadovanHUJAK - Hrvatska udruga Java korisnika / Croatian Java User AssociationPDF
Javantura v6 - On the Aspects of Polyglot Programming and Memory Management i...HUJAK - Hrvatska udruga Java korisnika / Croatian Java User AssociationPPTX
Javantura v6 - Case Study: Marketplace App with Java and Hyperledger Fabric -...HUJAK - Hrvatska udruga Java korisnika / Croatian Java User AssociationPDF
Javantura v6 - How to help customers report bugs accurately - Miroslav Čerkez...HUJAK - Hrvatska udruga Java korisnika / Croatian Java User AssociationPDF
Javantura v6 - When remote work really works - the secrets behind successful ...HUJAK - Hrvatska udruga Java korisnika / Croatian Java User AssociationPDF
Javantura v6 - Kotlin-Java Interop - Matej VidakovićHUJAK - Hrvatska udruga Java korisnika / Croatian Java User AssociationPDF
Javantura v6 - Spring HATEOAS hypermedia-driven web services, and clients tha...HUJAK - Hrvatska udruga Java korisnika / Croatian Java User AssociationPDF
Javantura v6 - End to End Continuous Delivery of Microservices for Kubernetes...HUJAK - Hrvatska udruga Java korisnika / Croatian Java User AssociationPPTX
Javantura v6 - Istio Service Mesh - The magic between your microservices - Ma...HUJAK - Hrvatska udruga Java korisnika / Croatian Java User AssociationPDF
Javantura v6 - How can you improve the quality of your application - Ioannis ...HUJAK - Hrvatska udruga Java korisnika / Croatian Java User AssociationPDF
Javantura v6 - Automation of web apps testing - Hrvoje RuhekHUJAK - Hrvatska udruga Java korisnika / Croatian Java User AssociationPDF
Javantura v6 - Master the Concepts Behind the Java 10 Challenges and Eliminat...HUJAK - Hrvatska udruga Java korisnika / Croatian Java User AssociationPDF
Javantura v6 - Building IoT Middleware with Microservices - Mario KusekHUJAK - Hrvatska udruga Java korisnika / Croatian Java User AssociationPDF
Javantura v6 - JDK 11 & JDK 12 - Dalibor TopicHUJAK - Hrvatska udruga Java korisnika / Croatian Java User AssociationJava cro'21 the best tools for java developers in 2021 - hujakHUJAK - Hrvatska udruga Java korisnika / Croatian Java User Association
JavaCro'21 - Java is Here To Stay - HUJAK KeynoteHUJAK - Hrvatska udruga Java korisnika / Croatian Java User Association
Javantura v7 - Behaviour Driven Development with Cucumber - Ivan LozićHUJAK - Hrvatska udruga Java korisnika / Croatian Java User Association
Javantura v7 - The State of Java - Today and Tomowwow - HUJAK's Community Key...HUJAK - Hrvatska udruga Java korisnika / Croatian Java User Association
Javantura v7 - Learning to Scale Yourself: The Journey from Coder to Leader -...HUJAK - Hrvatska udruga Java korisnika / Croatian Java User Association
JavaCro'19 - The State of Java and Software Development in Croatia - Communit...HUJAK - Hrvatska udruga Java korisnika / Croatian Java User Association
Javantura v6 - Java in Croatia and HUJAK - Branko Mihaljević, Aleksander RadovanHUJAK - Hrvatska udruga Java korisnika / Croatian Java User Association
Javantura v6 - On the Aspects of Polyglot Programming and Memory Management i...HUJAK - Hrvatska udruga Java korisnika / Croatian Java User Association
Javantura v6 - Case Study: Marketplace App with Java and Hyperledger Fabric -...HUJAK - Hrvatska udruga Java korisnika / Croatian Java User Association
Javantura v6 - How to help customers report bugs accurately - Miroslav Čerkez...HUJAK - Hrvatska udruga Java korisnika / Croatian Java User Association
Javantura v6 - When remote work really works - the secrets behind successful ...HUJAK - Hrvatska udruga Java korisnika / Croatian Java User Association
Javantura v6 - Kotlin-Java Interop - Matej VidakovićHUJAK - Hrvatska udruga Java korisnika / Croatian Java User Association
Javantura v6 - Spring HATEOAS hypermedia-driven web services, and clients tha...HUJAK - Hrvatska udruga Java korisnika / Croatian Java User Association
Javantura v6 - End to End Continuous Delivery of Microservices for Kubernetes...HUJAK - Hrvatska udruga Java korisnika / Croatian Java User Association
Javantura v6 - Istio Service Mesh - The magic between your microservices - Ma...HUJAK - Hrvatska udruga Java korisnika / Croatian Java User Association
Javantura v6 - How can you improve the quality of your application - Ioannis ...HUJAK - Hrvatska udruga Java korisnika / Croatian Java User Association
Javantura v6 - Automation of web apps testing - Hrvoje RuhekHUJAK - Hrvatska udruga Java korisnika / Croatian Java User Association
Javantura v6 - Master the Concepts Behind the Java 10 Challenges and Eliminat...HUJAK - Hrvatska udruga Java korisnika / Croatian Java User Association
Javantura v6 - Building IoT Middleware with Microservices - Mario KusekHUJAK - Hrvatska udruga Java korisnika / Croatian Java User Association
Javantura v6 - JDK 11 & JDK 12 - Dalibor TopicHUJAK - Hrvatska udruga Java korisnika / Croatian Java User Association
Recently uploaded (20)
Javantura v4 - Getting started with Apache Spark - Dinko Srkoč
- 1. Getting Started with Apache Spark and Scala Dinko Srkoč Instantor Technology Services
- 2. About Apache Spark (inevitable but hopefully quick intro) ● Started at UC Berkeley in 2009 ● General purpose cluster computing system ● Fast: 10x on disk, 100x in memory vs Hadoop MapReduce ● Runs locally, in the cloud, on Hadoop, Mesos ● High level APIs in: ○ Scala ○ Python ○ Java ○ R
- 3. About Apache Spark The Stack: ● SQL - SQL and semi/structured data processing ● MLLib - machine learning algorithms ● GraphX - graph processing ● Streaming - stream processing of live data streams
- 4. Data collections in Spark Collections: immutable, distributed, partitioned across nodes, operated in parallel ● Resilient Distributed Dataset (RDD) ○ Basic abstraction ○ Low-level API ○ Suitable for unstructured data (media, streams of text) ● Dataset/DataFrame ○ Dataset[T] - typed API, DataFrame (a.k.a. DataSet[Row]) - untyped API ○ High-level expressions: filters/maps, aggregations, averages, SQL queries, columnar access ○ optimizations
- 5. Demo The Menu: ● Starter - spark shell ○ Loading from different sources ○ The inevitable word count example ● Intermediate - spark notebook ○ Documentation, data visualization ● Main course - back to shell ○ streaming ○ Spark UI ● Dessert - mini project: ○ SBT ○ Deploying to Google Cloud Dataproc