Cloudera Data Science Workbench: sparklyr, implyr, and More - dplyr Interfaces to Large-scale Data
1 like997 views
Cloudera aims to empower data analysts and scientists to efficiently work with large-scale distributed data using tools like Apache Spark and Impala. The dplyr package facilitates common data manipulation tasks and translates commands for remote data sources into SQL, making it usable for both local and distributed environments. Key tips for effective use with SQL data sources include using show_query(), filtering early, checking data types, understanding your SQL engine, and knowing when to collect data.
![4© Cloudera, Inc. All rights reserved.
]
SQLPySpark
SparkR
SQL
SQL or
DataFrame API
SQL or
DataFrame API
SQL or
DataFrame API
SQL or
DataFrame API
SQL or
DataFrame API
SQL or
DataFrame API](https://image.slidesharecdn.com/clouderadatascienceworkbenchsparklyrimplyrandmoredplyrinterfacestolarge-scaledata3-180322045321/85/Cloudera-Data-Science-Workbench-sparklyr-implyr-and-More-dplyr-Interfaces-to-Large-scale-Data-4-320.jpg)
![6© Cloudera, Inc. All rights reserved.
]
SQLPySpark
SparkR
SQL
dplyr](https://image.slidesharecdn.com/clouderadatascienceworkbenchsparklyrimplyrandmoredplyrinterfacestolarge-scaledata3-180322045321/85/Cloudera-Data-Science-Workbench-sparklyr-implyr-and-More-dplyr-Interfaces-to-Large-scale-Data-6-320.jpg)
Cloudera Data Science Workbench: sparklyr, implyr, and More - dplyr Interfaces to Large-scale Data
- 1. 1© Cloudera, Inc. All rights reserved. dplyr Interfaces to Large-Scale Data Ian Cook @ianmcook [email protected]
- 2. 2© Cloudera, Inc. All rights reserved. Mission for Cloudera: Provide a platform for data analysts, data scientists to efficiently query, analyze, model large-scale data in clusters, cloud storage • By distributing Apache Spark, Apache Impala, other tools • By enabling productive use of these tools Python and R users often have difficulty moving from smaller data to large-scale distributed data • Familiar packages, methods don’t work the same way on distributed data Context
- 3. 3© Cloudera, Inc. All rights reserved. Poll question
- 4. 4© Cloudera, Inc. All rights reserved. ] SQLPySpark SparkR SQL SQL or DataFrame API SQL or DataFrame API SQL or DataFrame API SQL or DataFrame API SQL or DataFrame API SQL or DataFrame API
- 5. 5© Cloudera, Inc. All rights reserved. Poll question
- 6. 6© Cloudera, Inc. All rights reserved. ] SQLPySpark SparkR SQL dplyr
- 7. 7© Cloudera, Inc. All rights reserved. dplyr provides a set of verbs that perform common data manipulation steps • select() to select columns • filter() to filter rows • arrange() to order rows • mutate() to create new columns • summarise() to aggregate • group_by() to perform operations by group dplyr works on local data and with remote data sources • For remote sources, dplyr commands are translated into SQL dplyr
- 8. 8© Cloudera, Inc. All rights reserved. Poll question
- 9. 9© Cloudera, Inc. All rights reserved. Demonstration Example code at github.com/ianmcook/dplyr-examples
- 10. 10© Cloudera, Inc. All rights reserved. dplyr SQL backends dplyr ↕ dbplyr ↕ dplyr SQL backend package* ↕ DBI ↕ DBI-compatible interface package ↕ database driver or connector ↕ database/engine * optional
- 11. 11© Cloudera, Inc. All rights reserved. • Provides a SQL backend to dplyr for Spark • Also exposes the MLlib API and a subset of the Spark DataFrames API • Developed by RStudio spark.rstudio.com sparklyr
- 12. 12© Cloudera, Inc. All rights reserved. • Provides a SQL backend to dplyr for Impala • Uses ODBC or JDBC to connect to Impala • Developed at Cloudera tiny.cloudera.com/implyr implyr implyr
- 13. 13© Cloudera, Inc. All rights reserved. Five tips for using dplyr with SQL data sources
- 14. 14© Cloudera, Inc. All rights reserved. Use show_query() 1
- 15. 15© Cloudera, Inc. All rights reserved. filter() early arrange() late 2
- 16. 16© Cloudera, Inc. All rights reserved. Check your data types 3
- 17. 17© Cloudera, Inc. All rights reserved. Know your SQL engine 4
- 18. 18© Cloudera, Inc. All rights reserved. Know when to collect() 5
- 19. 19© Cloudera, Inc. All rights reserved. Questions? Ian Cook @ianmcook [email protected]
- 20. 20© Cloudera, Inc. All rights reserved. Cloudera Data Science Workbench More information tiny.cloudera.com/cdsw OnDemand training tiny.cloudera.com/cdsw-training