Analyzing Hadoop Data Using Sparklyr
6 likes2,457 views
The document discusses how Sparklyr allows data scientists to access and work with data stored in Cloudera Enterprise using the popular RStudio IDE. It describes the challenges data scientists face in accessing secured Hadoop clusters and limitations of notebook environments. Sparklyr integration with RStudio provides a familiar environment for data scientists to access Hadoop data and compute using Spark, enabling distributed data science workflows directly in R. The presentation demonstrates how to analyze over a billion records using Spark and R through Sparklyr.
![14© Cloudera, Inc. All rights reserved.
R for data science toolchain
“You’ll learn how to get your data into R [with Spark], get it into the
most useful structure, transform it, visualize it and model it.”](https://image.slidesharecdn.com/rstudiowebinarv21-170126195916/85/Analyzing-Hadoop-Data-Using-Sparklyr-14-320.jpg)
Analyzing Hadoop Data Using Sparklyr
- 1. 1© Cloudera, Inc. All rights reserved. Analyzing Hadoop Data with Sparklyr Accessing and working with data in Cloudera Enterprise through the popular RStudio IDE.
- 2. 2© Cloudera, Inc. All rights reserved. Your Speakers Nathan Stephens Director of Solutions Engineering (RStudio) Sean Anderson Data Science and Engineering (Cloudera)
- 3. 3© Cloudera, Inc. All rights reserved. Data Preparation Data Modeling Model Deployment (maybe) What does a Data Scientist Do?
- 4. 4© Cloudera, Inc. All rights reserved. Challenges in the data science process Data Engineering Data Science (Exploratory) Production (Operational) Data Wrangling Visualization and Analysis Model Training & Testing Production Model Preparation Batch Scoring Online Scoring Serving Dev Tools: IDEs/Notebooks, Collaboration Ops Tools: Versioning, Scheduling, Workflow, Publishing Data GovernanceGovernance Processing Acquisition Model Quality & Performance Experiments 1. Data scientists cannot easily access Hadoop data or compute using their favorite languages/frameworks. “Laptop data science” silos persist. 2. For IT, duplicate environments are costly and inconsistent, with limited security and governance. Wants to move to shared data and infrastructure.
- 5. 5© Cloudera, Inc. All rights reserved. Common Limitations Access Many times secured clusters are hard for data science professionals to connect either because they don’t have the right permissions or resources are to scarce to afford them access. In addition popular frameworks and libraries don’t read Hadoop data formats out-of-the- box. Scale Notebook environments seldom have large enough data storage for medium, let alone big data. Data scientists are often relegated to sample data and constrained when working on distributed systems. Popular frameworks and libraries don’t easily parallelize across the cluster. Developer Experience Popular notebooks don’t work well with access engines like Spark and package deployment and dependency management across multiple software versions is often hard to manage. Then once a model is built there is no easy path from model development to production
- 6. 6© Cloudera, Inc. All rights reserved. Cloudera’s Data Science Solution Familiar tools for distributed data science IDE Integration Interactive search and immediate exploration Search Audit, lineage, encryption, key management, & policy lifecycles Navigator Easy deployment and flexible scaling Cloud Deployment Modern Real-time Analytics Engine Spark Large-scale ETL & batch processing engine Hive-on- Spark Multi-Storage, Multi-Environment
- 7. 7© Cloudera, Inc. All rights reserved. Hadoop as a Data Science Platform • Leverage Big Data • Enable real-time use cases • Provide sufficient toolset for the Data Analysts • Provide sufficient toolset for the Data Scientists + Data Engineers • Provide standard data governance capabilities • Provide standard security across the stack • Provide flexible deployment options • Integrate with partner tools • Provide management tools that make it easy for IT to deploy/maintain Hadoop Kafka, Spark Streaming, Kudu Spark, Hive, Impala, Hue Partners Navigator + Partners Kerberos, Sentry, Record Service, KMS/KTS Cloudera Director Rich Ecosystem Cloudera Manager/Director
- 8. 8© Cloudera, Inc. All rights reserved. Our Goal: Bring more data science users to Hadoop Help more data scientists use the power of Hadoop Use a powerful, familiar environment with direct access to Hadoop data and compute Data Scientist Data Engineer Make it easy and secure to add new users, use cases Offer secure self-service analytics and a faster path to production on common, affordable infrastructure Enterprise Architect Hadoop Admin
- 9. 9© Cloudera, Inc. All rights reserved. R Studio • Integrated Development Environment for R (i.e. like Eclipse or Visual Studio but for R) • Runs on the Desktop (Windows, OSX, Linux) or over the Web as a server to enable shared resources and collaboration • Released in 2012 and now the most popular front-end for R users with thousands of downloads per day
- 10. 10© Cloudera, Inc. All rights reserved. RStudio Using R with Spark
- 11. 11© Cloudera, Inc. All rights reserved.
- 12. 12© Cloudera, Inc. All rights reserved.
- 13. 13© Cloudera, Inc. All rights reserved. What’s new with sparklyr 0.5
- 14. 14© Cloudera, Inc. All rights reserved. R for data science toolchain “You’ll learn how to get your data into R [with Spark], get it into the most useful structure, transform it, visualize it and model it.”
- 15. 15© Cloudera, Inc. All rights reserved. Understand Transform dplyr + sparklyr + SparkSQL Visualize ggplot2 Model sparklyr + ML Spark and R
- 16. 16© Cloudera, Inc. All rights reserved. Communicate R Markdown Notebooks R Markdown
- 17. 17© Cloudera, Inc. All rights reserved. Spark Deployment Modes Hadoop Yarn Driver Node Livy Standalone Cluster Local Apache Mesos EC2 SPARK EXECUTOR SPARK EXECUTOR YARN DRIVER LIVY RStudio Server RStudio Desktop
- 18. 18© Cloudera, Inc. All rights reserved. spark.rstudio.com
- 19. 19© Cloudera, Inc. All rights reserved. Demo Analyzing 1 billion records with Spark and R http://colorado.rstudio.com:3939/content/262/
- 20. 20© Cloudera, Inc. All rights reserved. Questions?