Extending your Hadoop Implementation to the Cloud
1 like•1,371 views
This document discusses strategies for extending Hadoop implementations to the cloud. It describes using infrastructure as a service (IaaS) to run Hadoop clusters in the cloud, leveraging cloud storage for flexibility, and using Hadoop as a service offerings. Hybrid scenarios involving both on-premises and cloud deployments are also covered. Demonstrations show deploying Hadoop clusters to Microsoft Azure and using Azure HDInsight.
Extending your Hadoop Implementation to the Cloud
- 1. EXTENDING YOUR HADOOP IMPLEMENTATION TO THE CLOUD Matt Winkler Principal Lead Program Manager Big Data @ Microsoft We’re Hiring @mwinkle
- 2. AGENDA Decisions IaaS Cloud Storage Hadoop as a Service Hybrid Scenarios Next Steps
- 3. WHY CLOUD? Elasticity Cost Optimization Economic flexibility Support for bursting workloads Global footprint
- 4. WHY ON-PREMISES? Compliance requirements Specific control over hardware/networking Integration requirements for additional apps to be close to cluster
- 5. WHEN CLOUD? Data born in the cloud Global apps Satisfy geopolitical or compliance constraints Dev/Test Backup Geo-Redundancy Bursting to cloud
- 6. IAAS – RUN YOUR HADOOP IN THE CLOUD IaaS offerings across the cloud providers offer: OS choice Node configuration Customized networking topology Repeatable, scriptable deployments You still have to: Set up the cluster Manage data movement into the cluster Integrate with your other applications Manage patching and updates of OS and apps Obtain support and/or licenses
- 7. DEMO Deploying a Hadoop Cluster to Azure
- 8. LEVERAGE CLOUD STORAGE FOR FLEXIBILITY Cloud storage enables economic flexibility, scale and rich features Size clusters independent of storage needs Clusters become stateless to operate across the data Price continues decreasing Geo-Redundancy allows for business continuity/disaster recover planning
- 9. CLOUD STORAGE USAGE PATTERNS HDFS within the cluster Move data in from cloud storage on boot (optional) backup/age data to cloud storage (optional) move data out to cloud storage to rebuild cluster Default file system using cloud storage connectors To Hadoop apps, they just see a path to data and most things “just work” Apps which rely specifically on HDFS may encounter compat issues The physics change in exchange for flexibility
- 10. LEVERAGING HADOOP AS A SERVICE Hadoop Services Cluster creation on demand Default integration with cloud storage Integration across services and apps Higher level abstractions API set for integrating into apps Azure HDInsight Clusters provisioned on top of Azure Blob storage Deploy clusters of any size Entire stack supported by Microsoft Azure Active Directory Service Bus Scheduler Multi-Factor Authentication Express Route Azure SQL Database Azure Web Site Some example services
- 11. DEMO Getting Started with HDInsight
- 12. HYBRID SCENARIOS Key scenarios Offsite backup Dev/Test Burst to Cloud The decision is not an XOR, it’s on-premises AND cloud
- 13. Microsoft Azure Azure Storage HDInsight (Hadoop) Hadoop cluster deployed to IaaS DEMO On-Premises Hadoop Cluster (HDP 2.1) Running on CentOS HDFS YARN Tez Hive MR Falcon
- 14. GETTING STARTED Get started in the cloud (getting started cards available @ the Microsoft booth and up here at the stage) Create an HDInsight cluster, or try out deploying a Hadoop cluster to Azure http://aka.ms/howtohdinsight
- 19. Data being landed into HDFS, on-prem
- 20. Syncing to blob store
- 21. New file @ 1:38pm
- 22. @ 2:01 pm
- 23. From HDI Cluster