R on Hadoop
Download as PPTX, PDF0 likes663 views
R is a programming language for statistical analysis and data science. While powerful, R has limitations when dealing with large datasets due to being single-threaded and RAM-limited. Hadoop is a framework for distributed processing of large datasets across clusters of machines using MapReduce. R can be used on Hadoop through approaches like streaming, RHadoop, and Spark. Streaming requires writing mapper and reducer scripts in R. RHadoop provides R APIs for Hadoop. The demo showed linear regression in R and on Hadoop for predictive analysis. Lessons were to do maximum preprocessing before R and consider performance for production versus exploratory analysis.
![Linear regression in R
df.p2 <- df[df$product_page == 'phone_2', ]
m <- lm(purchased ~ visited, data=df.p2)
summary(m)](https://image.slidesharecdn.com/ronhadoop-140620142020-phpapp02/85/R-on-Hadoop-8-320.jpg)
R on Hadoop
- 1. R ON HADOOP Kostiantyn Kudriavtsev Lviv Hadoop User Group, June 19, 2014
- 2. Agenda • What is R? • Linear Regression • R on Hadoop • Summary
- 3. What is R? Object-oriented and functional language for Stats, Math and Data Science created by statisticians with comprehensive data visualisation and statistical modelling capabilities; 5000+ (and grow) freely available specialised algorithms for finance, economics, genomics, linguistic and so on; 2M+ users with specialised domain skills; … but some drawbacks are: - limited by RAM - single thread
- 4. R development environment RStudio is de- facto standard IDE for R development and available in local or server mode. Might be used not only for coding, but also visualisation. Suitable to develop R solutions on top of Hadoop.
- 5. Apache Hadoop is an software framework that supports data- intensive distributed applications based on MapReduce algorithm (MR). Main idea: move computation to data. MR idea: - Map step: Map(k1,v1) → list(k2,v2) - Magic here (sort by k2, data transfer between nodes, etc) - Reduce step: Reduce(k2, list (v2)) → (k3, v3) What is Hadoop?
- 6. Linear regression Web-store might use linear regression to predict sales of goods or discover trends. sale(Product) ~ visitors(Product) Linear regression might be used here: sale = α * visitors + β
- 7. Linear regression in R df <- read.csv("Phone.csv", header=TRUE) qq <- qplot(visited,purchased,colour=product_page, data=df) qq + geom_smooth(method='lm', formula=y~x)
- 8. Linear regression in R df.p2 <- df[df$product_page == 'phone_2', ] m <- lm(purchased ~ visited, data=df.p2) summary(m)
- 9. R on Hadoop Several options: • Hadoop streaming • RHadoop • RHipe • RSpark • Oracle R Advanced Analytics for Hadoop • etc.
- 10. R Hadoop streaming Hadoop was mainly designed to use Java and provides comprehensive Java API. Other languages can be used through “Streaming API” Streaming API utilised standard input (reading) and standard output (writing) OS possibilities. It provides lightweight API for MapReduce in compare to Java API. Streaming requires writing two separate scripts (per mapper and reducer) in any language (Python, Ruby, R, C#, Go, OCalm, Lisp, etc)
- 11. R Hadoop streaming Streaming API drawbacks: ● while the inputs to the reducer are grouped by key, they are still iterated over line-by-line, and the boundaries between keys must be detected by the user ● no possibilities to utilize different mappers in one MapReduce job ● no possibilities to create different outputs from reducer ● counters update through stderr Additional disadvantage of implementing streaming in R: •strong output control for R functions, because they are “buzzy”, however only meaning data must be pushed
- 12. R Hadoop streaming: Mapper
- 13. R Hadoop streaming: Reducer
- 14. RHadoop RHadoop - set of libraries (written in R language) for R languages aim to facilitate using R languages with Hadoop streaming to develop MR jobs. So, it has general drawbacks for Hadoop streaming.
- 15. RHadoop RHadoop is still R through Hadoop Streaming Advantages compared to Streaming: ● don’t need to manage key change in Reducer ● don’t need to control functions output manually ● simple R API covers Streaming API ● R code can be run on local env/Hadoop without changes
- 16. Demo time
- 17. R on Hadoop in Real Life Several steps are required to achieve the goal: 1. Data ingestion 2. Data preparation 3. R processing 4. Postprocessing http://static.vroomgirls.com/website/wp-content/uploads/2011/09/Route66Road%C2%A9-Dmitry- Rogozhin.jpg
- 18. Learned Lessons R is slow… for million calculations it’s even slow with Hadoop! How to improve the speed? Rewrite flow - maximum preprocessing work before R step. Hadoop streaming supports mapper/reducer in different languages. Think twice. R is great for exploratory analysis and researches, but in production might cause performance penalty.
- 19. Q&A • Thank you for your attention