Migrating Big Data Workloads to the Cloud
- 1. www.clairvoyantsoft.com Migrating Big Data Workloads to the Cloud By: Robert Sanders
- 2. | 2 Robert Sanders Director of Big Data Engineering https://www.linkedin.com/in/robert-sanders-cs/ Robert Sanders is the Director of Big Data Engineering at Clairvoyant Insight. In his day job, Robert wears multiple hats and goes between leading members of the Insight development team to working directly with clients to Architecting and Engineering large scale Data projects. Robert has a deep background in enterprise systems, initially working on full-stack implementations and then focusing on building Data Management Platforms.
- 3. | 3 About Clairvoyant Background Awards & Recognition Boutique consulting firm centered on building data solutions and products All things Web and Data Engineering, Analytics, ML and User Experience to bring it all together Support core Hadoop platform, data engineering pipelines and provide administrative and devops expertise focused on Hadoop
- 4. | 4 ● “The Cloud” and “The Why” ● Cloud Migration Strategies ● Data Migration ● Application Migration Agenda
- 5. | 5 Rented Infrastructure and Services The Cloud
- 6. | 6 Reasons to Move to the Cloud ● Ability to make use of Cloud Services ● Reduce Cost ● Reduce Time to Spin Up new Instances ● Scalability
- 7. | 7 ● Repurchase (Drop and Shop) ● Rehost (Lift and Shift) ● Replatform (Lift, Tinker and Shift) ● Refactor/Re-Architect ● Retiring ● Retaining General Migration Strategies - 6 R’s
- 8. | 8 1. Forklift 2. Hybrid 3. Cloud Native Big Data Migration Strategies
- 9. | 9 ● Rehost Option Forklift
- 10. | 10 ● Rehost, Replatform and Refactor/Re-Architect Option Hybrid
- 11. | 11 Options with Blob Storage
- 12. | 12 ● Replatform and Refactor/Re-Architect Option Cloud Native
- 13. | 13 Forklift Hybrid Cloud Native
- 14. | 14 $ hadoop distcp hdfs://src_host/path/* hdfs://dest_host/path/ $ hadoop distcp hdfs://src_host/path/* s3a://dest_bucket/ Data Migration Strategies - DistCP
- 15. | 15 File Transfer Rate Note: Estimates (assumes - 25% network overhead) 100 Mbps 1 Gbps 10 Gbps 1 TB 30 hrs 3 hrs 18 min 10 TB 12 days 30 hrs 3 hrs 100 TB 124 days 12 days 30 hrs 1 PB 3 years 124 days 12 days 10 PB 34 years 3 years 124 days Usable Network Bandwidth Data to Transfer
- 16. | 16 ● Azure ExpressRoute Data Migration Strategies - DistCP
- 17. | 17 ● Use an External Appliance ○ AWS Snowball (50 or 80 TB of data) ○ AWS Snowmobile (20+ PB Data) ● Loads Data into S3 ● Then copy Deltas from On-Prem to Cloud Data Migration Strategies - External Appliance
- 18. | 18 ● S3 Transfer Acceleration ● AWS Snowball (50 or 80 TB of data) ● AWS Snowmobile (20+ PB Data) ● AWS Direct Connect ● AWS DataSync ● AWS Transfer for SFTP ● Amazon Kinesis Firehose ● More Data Migration Services ● Azure Data Factory ● Azure Database Migration Service ● Azure ExpressRoute Amazon Azure
- 19. | 19 On-Prem Workloads Time 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 Daily Reporting Analytics Real-Time MLTraining Higher Utilization Lower Utilization
- 20. | 20 Cloud Native Workloads
- 21. | 21 ● Batch ○ Amazon EMR (MapReduce, Hive, Pig, Spark), Amazon Redshift, AWS Glue ● Interactive ○ Amazon Redshift, Amazon Athena, Amazon EMR (Presto, Spark, Hive) ● Stream ○ Amazon EMR (Spark Streaming, Flink, Storm), Amazon Kinesis Analytics ● AI ○ Amazon AI (Lex, Polly, ML, Rekognition), Amazon EMR (Spark ML), Deep Learning) Workload Migration to be Cloud Native