Sawmill - Integrating R and Large Data Clouds
Download as PPTX, PDF1 like1,262 views
This document discusses using R for large-scale data analysis on distributed data clouds. It recommends splitting large datasets into segments using MapReduce or UDFs, then building separate models for each segment in R. PMML can be used to combine the separate models into an ensemble model. The Sawmill framework is proposed to preprocess data in parallel, build models for each segment using R, and combine the models into a PMML file for deployment. Running R on each segment sequentially allows scaling to large datasets, with examples showing processing times for different numbers of segments.
Sawmill - Integrating R and Large Data Clouds
- 1. SawmillSome Lessons Learned Running R in Large Data CloudsRobert GrossmanOpen Data Group
- 2. What Do You Do if Your Data is to Big for a Database?Give up and invoke sampling.Buy a proprietary system and ask for a raise.Begin to build a custom system and explain why it is not yet done.Use Hadoop.Use an alternative large data cloud (e.g. Sector)
- 3. Basic IdeaTurn it into a pleasantly parallel problem.Use a large data cloud to manage and prepare the data.Use a Map/Bucket function to split the job.Run R on each piece using Reduce/UDF or streams.Use PMML multiple models to glue the pieces together.
- 4. Why Listen?This approach allows you to scale R relatively easily to hundreds of TB to PB.The approach is easy.(A plus: it may look hard to your colleagues, boss or clients.)There is at least an order of magnitude of performance to be gained with the right design.
- 5. Part 1. Stacks for Big Data5
- 6. The Google Data StackThe Google File System (2003)MapReduce: Simplified Data Processing… (2004)BigTable: A Distributed Storage System… (2006)6
- 7. Map-Reduce ExampleInput is file with one document per recordUser specifies map functionkey = document URLValue = terms that document contains“it”, 1“was”, 1“the”, 1“best”, 1(“doc cdickens”,“it was the best of times”)map
- 8. Example (cont’d)MapReduce library gathers together all pairs with the same key value (shuffle/sort phase)The user-defined reduce function combines all the values associated with the same keykey = “it”values = 1, 1“it”, 2“was”, 2“best”, 1“worst”, 1key = “was”values = 1, 1reducekey = “best”values = 1key = “worst”values = 1
- 9. Applying MapReduce to the Data in Storage Cloudshuffle/reducemap9
- 10. Google’s Large Data CloudCompute ServicesData ServicesStorage Services10ApplicationsGoogle’s MapReduceGoogle’s BigTableGoogle File System (GFS)Google’s Stack
- 11. Hadoop’s Large Data CloudApplicationsCompute Services11Hadoop’sMapReduceData ServicesNoSQL DatabasesHadoop Distributed File System (HDFS)Storage ServicesHadoop’s Stack
- 12. Amazon Style Data CloudLoad BalancerSimple Queue Service12SDBEC2 InstanceEC2 InstanceEC2 InstanceEC2 InstanceEC2 InstanceEC2 InstanceEC2 InstanceEC2 InstanceEC2 InstanceEC2 InstanceEC2 InstancesEC2 InstancesS3 Storage Services
- 13. Sector’s Large Data Cloud13ApplicationsCompute ServicesSphere’s UDFsData ServicesSector’s Distributed File System (SDFS)Storage ServicesRouting & Transport ServicesUDP-based Data Transport Protocol (UDT)Sector’s Stack
- 14. Apply User Defined Functions (UDF) to Files in Storage Cloudmapshuffle /reduce14UDFUDF
- 15. Folklore MapReduce is great.But sometimes it is easier to use UDFs or other parallel programming frameworks for large data clouds.And often it is easier to use Hadoop streams, Sector streams, etc.
- 17. Terasort BenchmarkSector/Sphere 1.24a, Hadoop 0.20.1 with no replication on Phase 2 of Open Cloud Testbed with co-located racks.
- 19. MalStoneSector/Sphere 1.20, Hadoop 0.18.3 with no replication on Phase 1 of Open Cloud Testbed in a single rack. Data consisted of 20 nodes with 500 million 100-byte records / node.
- 20. Part 2Predictive Model Markup Language
- 21. Problems Deploying Models Models are deployed in proprietary formats Models are application dependent Models are system dependent Models are architecture dependant Time required to deploy models and to integrate models with other applications can be long.
- 22. Predictive ModelMarkup Language (PMML) Based on XML Benefits of PMMLOpen standard for Data Mining & Statistical Models Not concerned with the process of creating a modelProvides independence from application, platform, and operating systemSimplifies use of data mining models by other applications (consumers of data mining models)
- 23. PMML Document ComponentsData dictionaryMining schemaTransformation DictionaryMultiple models, including segments and ensembles.Model verification, …Univariate Statistics (ModelStats)Optional Extensions
- 24. PMML Models polynomial regression logistic regression general regression center based clusters density based clusterstrees
- 25. associations
- 26. neural nets
- 27. naïve Bayes
- 28. sequences
- 29. text models
- 30. support vector machines
- 31. rulesetPMML Producer & Consumers25Modeling Environment211Model ProducerDataData Pre-processingPMMLModelDeployment Environment 2PMMLModel331Model ConsumerPost Processingdataactionsscoresrules
- 32. Part 3Sawmill
- 33. Step 2: Invoke R on each segment/bucket and build PMML model Step 1: Preprocess data using MapReduce or UDFmodelsStep 3: Gather the models together to form a multiple model PMML file
- 34. Step 1: Preprocess data using MapReduce or UDFStep 2: Build separate model in each segment using RStep 1: Preprocess data using MapReduce or UDFStep 2: Score data in each segment using R
- 35. Sawmill SummaryUse HadoopMapReduce or Sector UDFsto preprocess the dataUse HadoopMap or Sector buckets to segment the data to gain parallelismBuild separate statistical model for each segment using R & Hadoop / Sector StreamsUse multiple models specification in PMML version 4.0 to specify segmentationExample: use Hadoop Map function to send all data for each web site to different segment (on different processor)
- 36. Small Example: Scoring Engine written in RR processed a typical segment in 20 minutes
- 37. Using R to score 2 segments concatenated together = 60 minutes
- 38. Using R to score 3 segments concatenated together = 140 minutesWith Sawmill Framework1 month of data, about 50 GB, hundreds of segments
- 39. 300 mapper keys / segments
- 40. Mapping and Reducing < 2 minutes
- 41. Scoring: 20 minutes * max of segments per reducer
- 42. Had anywhere from 2 to 3 reducers per node and 2 to 8 segments per reducer.
- 43. Often ran in under 2 hours.Reducer R Process?There are at least three ways to tie theMapReduceprocess to the R process.
- 44. MACHINE: One instance of the R process on each data node (ornper node)
- 45. REDUCER: One instance of the R process bound to each reducer
- 46. SEGMENT: Instances can be launched by the reducers as necessary (when keys are reduced)TradeoffsYou need to have a general idea of
- 47. how long the records for a key take to be reduced.
- 48. how long the application takes to process the segment
- 49. how many keys are seen per reducer
- 50. In order to prevent bottlenecksThank You!www.opendatagroup.com