Backup automation in KAKAO
1 like656 views
This document discusses strategies for automating remote database backups across multiple data centers. It recommends scheduling backups automatically after a queue time to use underutilized backup servers. The backup manager would select the target backup server based on its service zone, data center location, and available quota to balance load. It would also avoid using the same backup server consecutively and start backups at different times each day to improve reliability in case of failures.
Backup automation in KAKAO
- 1. Backup Automation SUNG DONGCHAN From KAKAO
- 3. 2. Replication 1. Recovery 3. etc...
- 5. 2. xtrabackup 1. mysqldump 3. snapshot
- 7. Local Disk Easy, No scheduling But.. Recovery?
- 8. Remote Storage Network Traffic? Rare affect on Recovery NIC
- 10. • Schedule with CRONTAB • Start from manager server on DEFINED TIME (BUT, Everyday it’s same time..) Same time
- 11. No Problem if it's on LOCAL BACKUP
- 13. A lot of Servces
- 18. TODAY!! I’ll talk about remote backup scheduling
- 19. Backup Files? • Big.. but Rarely Used • Transaction Log exists 2~3 Days
- 20. THEN!!!!
- 21. MUST "Same Backup Server" As Yesterday? Question 1. NO
- 22. Question 2. MUST "Today's Backup File"? NO
- 23. Question 3. MUST “Backup on Same Time"? NO
- 24. Think about.. Backup File is in Cloud A B C D E F
- 25. Today Yesterday The day before yesterday Everyday, Different
- 26. Let's start backup on Unhurried Server
- 27. Consists of.. • Service Configuration • Backup Servers • Backup History
- 28. Service Configuration Queue Time? Backup job could be started ANY TIME after QUEUE TIME
- 29. Backup Server Quota? 0/3 3/3 1/3 Quota Full I'm So Busy!!
- 30. Backup Server Located on “Specific RACK”
- 31. But.. Database could be on.. “Different” IDC!!
- 33. Service Zone!! Virtual Service Group ex) Common Zone, Security Zone, ThirdParty Zone..
- 34. IDC1 IDC2 IDC3 Common Service Zone Security Zone ThirdParty Zone Service Zone
- 35. Backup History Today? Not Yesterday Server!! BACKUP001.kr Yesterday NOT!! BACKUP001.kr so.. BACKUP001.kr BACKUP002.kr BACKUP003.kr Service Candidate Today's Backup Server
- 36. Backup Step • Choose Service base on Queue Time • Choose Target Backup Server (Service Zone + IDC Location + Quota) • Start Backup
- 37. Choose Service (1/3) SVC01 SVC02 SVC03 SVC0400:00 Smaller Queue Time, Backup First 01:59 02:00 02:00 03:00
- 38. Choose Service (2/3) SVC01 SVC02 SVC03 SVC0400:00 Bigger Data File, Backup First 02:00 02:00 02:00 02:00 500G 300G 200G 100G
- 39. Choose Service (3/3) SVC01 SVC02 SVC03 SVC0400:00 Different Switch, Backup First 02:00 02:00 02:00 02:00 100G 100G 100G 100G SW01 SW02 SW03 SW01
- 40. Choose Backup Server (1/3) Target Backup Server List? • Same Service Zone • Same IDC
- 41. Choose Backup Server (2/3) Less than Quota 0/3 3/3 1/3 Quota Full I'm So Busy!! First Candidate
- 42. Choose Backup Server (3/3) Not Used Yesterday Backuped Yesterday Candidate Today's Backup Server
- 44. Start Backup At Service after queue time on Unhurried and Valid Backup Server Backup ServerService Zone
- 45. Backup Service Architect Databases Zone1 Databases Databases Zone2 Databases Manager Schedule Automatically Backup Servers Backup Servers Backup Servers Backup Servers IDC1 IDC2
- 46. Backup Timeline
- 47. Restore Sample ssh root@BACKUP_SERVER 'cat /data/backup/SERVICE_HOST/DATE.xbs' | xbstream -x ssh root@backup001 'cat /data/backup/svc01/20150427.xbs' | xbstream -x Generated By Manager
- 48. With LZOP? Recommand! If InnoDB :) Compressed
- 49. After Queue Time Same IDC Same Service Zone Unhurried Backup Server Backup Scheduled..
- 50. Q/A