Case Study: Implementing Hadoop and Elastic Map Reduce on Scale-out Object Storage
Download as PPTX, PDF4 likes1,659 views
This document discusses implementing Hadoop and Elastic MapReduce on Cloudian's scale-out object storage platform. It describes Cloudian's hybrid cloud storage capabilities and how their approach reduces costs and provides faster analytics by analyzing log and event data directly on their storage platform without needing to transform the data for HDFS. Key benefits highlighted include no redundant storage, scaling analytics with storage capacity by adding nodes, and taking advantage of multi-core CPUs for MapReduce tasks.
Case Study: Implementing Hadoop and Elastic Map Reduce on Scale-out Object Storage
- 1. Cloudian® S3 Cloud Storage Platform Case Study: Implementing Hadoop and Elastic Map Reduce on Scale-out Object Storage Paul Turner Cloudian Inc. June 11th 2014
- 2. About Cloudian • Hybrid cloud storage startup in Silicon Valley – Strong venture backing: Goldman Sachs, Intel Capital – Solid management with storage, big data, enterprise software and telco expertise – 50 employees, offices in Foster City, Japan and China • Production hardened product • Target market: mid- to large-enterprises & regional service providers • GTM: traditional storage distribution/VARs CLOUDIAN PARTNERS
- 3. The Challenge • Business problem = Analysis of log data from our customer systems to improve support (classic ‘Internet of Things’ content) • Existing system required transformation of the data into HDFS for analytics (slow and costly) Goal : Reduce cost and provide faster results 6/16/2014 3
- 4. Use Case : Support Analytics • Compare system statistics and usage patterns to previous normal results 6/16/2014 4 Abnormal Operations Analysis End User Analysis to root cause issues Trend Analysis for Capacity Planning and Traffic Patterns • Identify all operations for a particular user and review patterns and any faults • Build capacity and traffic trend lines based on statistical analysis of all traffic 100tps S3 Server = 83million lines info log = 3.5GB/Day 10 Server System = 35GB/Day ~ 1TB/month 100 Customer Systems => 1.2PB Annually
- 5. Traditional Big Data Flow Event Processing Platform Big Data Storage Platform Analytics PlatformContent Storage Consumer Activity (Events, GPS, WiFi) Social MediaDevice Tracking and Logs (Event, Configuration, Usage, Performance, ) Real Time Events Big Data Result of analysis 6/16/2014 5
- 6. Traditional Big Data Flow Event Processing Platform Analytics Platform (HDFS)Content Storage (Object, NAS) • Wasted storage = storage for content and analytics • Transform of data into HDFS can be costly • High overhead of HDFS (3copy replica) for content which may be poor quality Logs, Config 6/16/2014 6
- 7. S3 and Hadoop • Apache Hadoop supports S3 since Jan 2008 – http://wiki.apache.org/hadoop/AmazonS3 • Well-proven by Amazon with Elastic MapReduce • State-of-the-art and advancing quickly to provide much easier Hadoop over S3 – e.g. Netflix Genie – https://github.com/Netflix/genie 6/16/2014 7
- 8. Cloudian Approach Event Processing Platform AnalyticsCloudian HyperStore Storage • No redundant storage of data • Hyperstore scales out with your data – adding nodes for I/O • Analyze more - allows for efficient bulk data analysis in place • Take advantage of multi-core CPUs – makes sense for MapReduce • Can feed smarter data for subsequent analytic systems • Faster time to decision 6/16/2014 8
- 9. Cloudian Hadoop Configuration • Hadoop 2.2 • Configured for native S3 file system (etc/hadoop/core-site.xml) – S3N native file system for reading and writing regular files on S3. The advantage of this file system is that you can access files on S3 that were written with other tools. Conversely, other tools can access files written using Hadoop. • Configure Hadoop to use Cloudian (etc/hadoop/jets3t.properties) – s3service.s3-endpoint=CLOUDIAN_ENDPOINT – s3service.s3-endpoint-http-port=CLOUDIAN_PORT 6/16/2014 9 Note: you can also dedicate a bucket for Hadoop analytics and then Hadoop will chunk the content into blocks for storage – like HDFS
- 10. S3 NFS Cloudian HyperStore® Software Scalable peer-to-peer architecture Multi-data center replication Multi-Tenancy and Chargeback Hybrid cloud-ready (any S3 cloud) 100s of supported applications Optimized for any workload Storage for OpenStack & CloudStack 6/16/2014 10
- 11. Elastic, Distributed and Reliable NOSQL database distributes and replicates data Logical Ring Data is automatically replicated to multiple nodes. Location of data can be designated, for instance, to multiple datacenters and per rack. DC1 DC2 In theory, # of nodes in a logical ring can be up to 2127 (almost infinite). Data load can be rebalanced when a node is added or removed. Jun-14 116/16/2014
- 12. Enhanced HyperStore® Technology • Policies tailored for different object types • Optimized for all data • Chunking for better performance • Erasure Coding for deep archive efficiency • Reliable storage across multi-node failures HyperStore Patent Pending Small Objects Large Objects Active Content File System NOSQL DB Erasure Coding Deep Archives 6/16/2014 12
- 13. Cloudian Complete S3 API • Core REST API – Get, Put, Post, Head, Delete • Multi-part uploads: Allows uploading large objects in multiple parts • Versioning: Multiple versions of same object • Bucket Lifecycle: Auto-expiration using rules • Server side encryption: Managed by Cloudian • Location Constraint: Assign data to specific region (e.g. for HIPAA compliance) • Bucket Website: Create buckets as websites to host web content • Access control lists (ACLs) define access rights to bucket and object • And more... Cloudian Complete S3 API Products S3 API Cloudian AmpliData Basho Caringo Cleversafe EMC Atmos NetApp Bycast Scality OpenStack Swift 6/16/2014 13
- 14. Seamless tiering to Amazon S3, Glacier and other S3 Service Providers 146/16/2014 • Cloudian deployed as On-Premises S3 cloud behind the firewall • Automatically migrates data to AWS using Bucket Lifecycle Policies – Optional migration to Glacier – Metadata maintained for search/list of objects • Configurable to reduce overhead • Read/Writes to migrated objects – restore by default, option to redirect to AWS/S3 Service Provider On-Premises S3 S3 Client/Application Content migrated or restored via Bucket Lifecycle Policies Option to redirect migrated content Amazon S3 Firewall Amazon Glacier
- 15. Big Data Storage Platform 15 Event Processing Platform Big Data Storage Platform Input I/F Recommend CEP Engine Filter Judge Aggregate Real Time Analysis Big Data Analysis Analyze Recommend Data Analysis and Storage Platform Content Storage Consumer Activity (Events, GPS, WiFi) Social mediaBusiness Tracking (goods, inventory, campaign, sales) Smarter Business 6/16/2014
- 16. Future Work • Delivery of Cloudian Hadoop-ready object storage (2HCY14) • Integration with key Hadoop distributions • Locality awareness • Potentially use new drive technology for processing (eg HGST Ethernet drive) • Find out more – Booth 139 6/16/2014 16
- 17. Cloudian® S3 Cloud Storage Platform Thank You! Questions? www.cloudian.com “The Leading Provider of Hybrid Cloud Storage”