Using SparkR to Scale Data Science Applications in Production. Lessons from the Field: Spark Summit East talk by Heiko Korndorf
6 likes1,474 views
The document discusses scaling R applications in production using SparkR, highlighting the integration of data science and engineering within data pipelines. It covers various aspects including architecture, parallelization, and the use of Spark APIs, while presenting challenges and solutions encountered in the process. Additionally, it emphasizes the growing ecosystem around Spark and R, citing the importance of seamless collaboration between the two for effective data processing and analysis.
![Parallelization with SparkR 2.x
Support for User-Defined Functions
• dapply (dapplyCollect)
• input: DataFrame, func [, Schema]
• output: DataFrame
• gapply (gapplyCollect)
• input: DataFrame¦GroupedData,
groupBy, func [, Schema]
• output: DataFrame
• spark.lapply
• input: parameters, func
• Access to data/HDFS
• output: List](https://image.slidesharecdn.com/2302heikokorndorf-170215231735/85/Using-SparkR-to-Scale-Data-Science-Applications-in-Production-Lessons-from-the-Field-Spark-Summit-East-talk-by-Heiko-Korndorf-16-320.jpg)
Using SparkR to Scale Data Science Applications in Production. Lessons from the Field: Spark Summit East talk by Heiko Korndorf
- 1. Scaling SparkR in Production. Lessons from the Field. Heiko Korndorf Wireframe, CEO & Founder
- 2. About me Heiko Korndorf • CEO & Founder Wireframe • MS in Computer Science • Application Areas: ERP, CRM, BI, EAI • Helping companies in • Manufacturing • Telecommunications • Financial Services • Utilities • Oil & Gas • Professional Services Rapid Application Development for Hadoop/Spark Data Science-as-a-Service
- 3. What we’ll talk about Classify this talk …. • Data Science: Scaling your R application with SparkR • Data Engineering: How to bring Data Science applications into your production pipelines, i.e. adding R to your toolset. • Management: Integrating Data Science and Data Engineering with SparkR
- 4. Agenda • SparkRArchitecture 1.x/2.x • Reference Projects I + II • Approach with Spark 1.5/1.6 • Parallelization via YARN • Dynamic R Deployment, incl. dependencies/packages • Approach with Spark 2.0 • Parallelization via SparkR • R-Graphics: headless environment, concurrency • Use Spark APIs: SQL, Mllib • On-Prem vs Cloud (Elasticity/decouple storage and compute) • Integrating Data Science and Data Engineering • A Broader Look at the Ecosystem • Outlook and Next Steps
- 5. Data Science with R • Very popular language • Designed by statisticians • Large community • > 10.000 packages • plus: integrated package management • But: Limited as Single-Node platform • Data has to fit in memory • Limited concurrency for processing
- 7. SparkR as seen from R • Import SparkR-package and initialize SparkSession • Convert data frames from local R data frames to Spark DataFrame and back • Read and write data stored in Hadoop HDFS, HBase, Cassandra, and more • Use Spark Libraries, such as SparkSQL and ML • User cluster hardware to distribute data frames and parallelize computation
- 8. SparkR Architecture • Execute R on cluster • Data Integration • Spark DataFrame – R data frame • Access Big Data File Formats • Parallelization with UDFs • Use Spark APIs • SparkSQL • Spark MLlib
- 9. SparkSQL from R • Execute SQL against Spark DataFrame • SELECT • Specify Projection • WHERE • Filter criteria • GROUPBY • Group/Aggregate • JOIN • Join tables
- 10. Native Spark ML
- 11. Time Series Forecasting • ARIMA(p,d,q) • AR: p = order of the autoregressivepart • I: d = degree of first differencing involved • MA: q = order of the moving average part • Time Series: a series of data points indexed in time order • Methods: • ExponentialSmoothing • Neural Networks • ARIMA:
- 12. “Pedestrian” Challenges • Modify some Spark and R (custom-build) • Submit Spark job with R (incl. packages) as YARN dependency • Challenge: R not installed on cluster • R’s installation location is hard-coded in R • “R Markdown” produces HTML, PDF, and more • Best way to manage those outputs? • Producing additional output during run • Creating graphics in headless environments Installing R (+Pkg’s) on cluster Creating PNGs, HTML, PDF, …
- 13. Parallelization with SparkR 1.x • Sequentialcomputation:> 20 hrs. • Single-Server, parallelized:> 4.5 hrs
- 14. Parallelization with SparkR 1.x • Sequentialcomputation:> 20 hrs. • Single-Server, parallelized:> 4.5 hrs • SparkR 1.6.2, 25 nodes,4 cores: ca. 12 mins.
- 15. Microsoft R Server for Spark • Microsoft R Server for HDInsight integrates Spark and R • Based on Revolution Analytics • UDFs via rxExec() • Data Sources • RxXdfFile • RxTextFile • RxHiveData • RxParquetData
- 16. Parallelization with SparkR 2.x Support for User-Defined Functions • dapply (dapplyCollect) • input: DataFrame, func [, Schema] • output: DataFrame • gapply (gapplyCollect) • input: DataFrame¦GroupedData, groupBy, func [, Schema] • output: DataFrame • spark.lapply • input: parameters, func • Access to data/HDFS • output: List
- 18. The (Data) Science Process Public Perception of Science Source: Birth of a Theorem – with Cedric Villani (https://www.youtube.com/watch?v=yYwydG_aHPE)
- 19. The (Data) Science Process Public Perception of Science Science in Reality Source: Birth of a Theorem – with Cedric Villani (https://www.youtube.com/watch?v=yYwydG_aHPE)
- 20. Integrating Dev and Prod • No Need to Re-Write Applications for Production • Common Environmentfor Development,Testand Production • “Looks like R to Data Science, looks like Spark to Data Engineers”
- 21. 2-Level Parallelization (1) Submit multiple jobs to your cluster: - Cluster Manager (YARN, Spark, Mesos) - Spark Job: Driver and Executors (2) Use GPGPU - Spark Job: Driver and Executor - Let Executor use GPGPU (3) Combine 1 and 2
- 22. Mix Scala and R • Call R from Scala • Add DataScience Module to your Spark Application • Use Spark/Scala for ETL, R for Science code • Call Spark from R • Implement high-performance code in Spark • More granularcontrol over cluster resources
- 23. Spark & R: A Dynamic Ecosystem Hadoop, Spark & R: Many interesting projects and options • SparkR (Apache, Databricks) • R Server for Spark (Microsoft) • Sparklyr (RStudio) • SystemML (IBM) • FastR (Oracle) • Renjin (BeDataDriven)
- 24. Outlook & Misc • Organizational: Deepen Integration of Data Engineering & Data Science • Source Code Control & Versioning (git …) • Continuous Build • Test Management (RUnit, testthat…?) • Multi-Output (Rmarkdown) • Technical: New Approaches • Simplify/Unify Data Pipelines (SparkSQL) • Performance Improvement: use MLlib • Performance Improvement: move calculation to GPU