Database Migration to AWS at scale by Sameer
Download as PPTX, PDF•0 likes•86 views
The document discusses a technical meetup on MySQL migration to AWS led by Sameer Kumar, a DevOps engineer with over 10 years of experience. It covers the reasons for migrating databases to the cloud, various migration approaches, architecture for scalability and availability, and performance statistics related to the migration process. Detailed methods and tools for backup and restoration, such as mysqldump and Percona XtraBackup, are also presented along with deployment strategies like blue/green deployment.
Database Migration to AWS at scale by Sameer
- 1. MySQL Migration To AWS As delivered in tech meetup #01 organized by www.edYoda.com on 22nd April 2018 at zekeLabs Bangalore
- 2. About me: Sameer Kumar DevOps Engineer @ Intuit 10+ years Ex- Siemens, Ex- Paytm, Ex - Snapdeal
- 3. Why Migrate Databases to Cloud ?
- 4. Why Migrate Databases to Cloud ? Business Growth
- 5. Why Migrate Databases to Cloud ? Business Growth Vertical Scaling
- 6. Why Migrate Databases to Cloud ? Business Growth Horizontal ScalingVertical Scaling
- 7. Availability Zone 3 POD-1 POD-2 POD-N Proxy Layer (Routing) Availability Zone 1 Proxy Layer (Routing) POD-1 POD-2 POD-N Availability Zone 2 Proxy Layer (Routing) POD-1 POD-2 POD-N Architecture – Scalability & Availability
- 8. Some statistics ~700K Login per day, ~1MM users login per day during Peak seasons. 103TB of financial data hosted in 118 MySQL physical servers. Size varies from 600 GB to 2 TB Nearly Zero Down Time
- 10. Different Approaches Different Time 2015 Logical Backup Restore mysqldump
- 11. Different Approaches Different Time 2016 DMS 2015 Logical Backup Restore mysqldump
- 12. Different Approaches Different Time 2017 Percona Backup 2016 DMS 2015 Logical Backup Restore mysqldump
- 13. Logical Backup Restore • mysqldump • rsync • tar On-Premise Slave EC2 with EBS RDS • MySQL Client • tar
- 14. Logical Backup Restore • mysqldump • rsync • tar On-Premise Slave EC2 with EBS RDS • MySQL Client • tar • Dump : 18 Hours for 2 TB • Sync to AWS: 180G Per Hour • Restoration: 24~30 Hours • Replication Catchup : 2 Days
- 15. Logical Backup Restore • mysqldump • rsync • tar On-Premise Slave EC2 with EBS RDS • MySQL Client • tar • Dump : 18 Hours for 2 TB • Sync to AWS: 180G Per Hour • Restoration: 24~30 Hours • Replication Catchup : 2 Days • max_allowed_packet • No MyISAM • Ignore MySQL schema • Don’t change user permission during migration
- 16. Percona XtraBackup • Percona Xtrabackup • AWS CLI • tar On-Premise Slave AWS S3 RDS
- 17. Percona XtraBackup • Percona Xtrabackup • AWS CLI • tar On-Premise Slave AWS S3 RDS • Only MySQL 5.6 • 6 TB Limit • Functions, Stored Procedures are not imported automatically • Can’t use Partial Backup
- 18. Percona XtraBackup • Percona Xtrabackup • AWS CLI • tar On-Premise Slave AWS S3 RDS • Only MySQL 5.6 • 6 TB Limit • Functions, Stored Procedures are not imported automatically • Can’t use Partial Backup • S3 Copy: 180G Per Hour • Restoration: 8 Hours for 2 TB • Replication Catchup : 2 Days
- 19. Blue/Green Deployment with DNS switch DB-1 DB-2 DB-N Availability Zone 1 Proxy Layer (Routing & Pod-picking) App-1 App-2 App-N Availability Zone 3 Proxy Layer (Routing & Pod-picking) App-1 App-2 App-N Availability Zone 2 Proxy Layer (Routing & Pod-picking) App-1 App-2 App-N Route 53 Persistence Layer - RDS Jenkins Web and Application Layer Availability Zone 1 Proxy Layer (Routing & Pod-picking) App-1 App-1 App-1 Availability Zone 3 Proxy Layer (Routing & Pod-picking) App-1 App-2 App-N Availability Zone 2 Proxy Layer (Routing & Pod-picking) App-1 App-2 App-N Web and Application Layer 1. Deploy 2. Validate 3. Switch DNS Entry
- 21. Reference links • DMS Limitations • Migration to RDS using Percona • RDS as Slave of External MySQL
- 22. Thank You