MongoDB World 2019: Modern Data Backup and Recovery from On-premises to the Public Cloud
1 like202 views
The presentation discussed MongoDB's approach to modern data backup and recovery from on-premises to the public cloud. It introduced WiredTiger checkpoints that take checkpoints of data and move them to long-term storage, eliminating the need for initial syncs and reducing infrastructure requirements. It also discussed Atlas' multi-AZ disaster recovery, point-in-time data recovery with queryable backups, and continuous backup regions using cloud provider snapshots to satisfy data sovereignty at a lower cost. Future improvements may allow point-in-time restores with customizable windows and building new clusters from backups.
MongoDB World 2019: Modern Data Backup and Recovery from On-premises to the Public Cloud
- 1. Benjamin Cefalo, Lead Product Manager, MongoDB Modern Data Backup and Recovery from On Premises to the Public Cloud @bencefalo
- 2. Safe Harbor Statement This presentation contains “forward-looking statements” within the meaning of Section 27A of the Securities Act of 1933, as amended, and Section 21E of the Securities Exchange Act of 1934, as amended. Such forward-looking statements are subject to a number of risks, uncertainties, assumptions and other factors that could cause actual results and the timing of certain events to differ materially from future results expressed or implied by the forward-looking statements. Factors that could cause or contribute to such differences include, but are not limited to, those identified our filings with the Securities and Exchange Commission. You should not rely upon forward-looking statements as predictions of future events. Furthermore, such forward-looking statements speak only as of the date of this presentation. In particular, the development, release, and timing of any features or functionality described for MongoDB products remains at MongoDB’s sole discretion. This information is merely intended to outline our general product direction and it should not be relied on in making a purchasing decision nor is this a commitment, promise or legal obligation to deliver any material, code, or functionality. Except as required by law, we undertake no obligation to update any forward-looking statements to reflect events or circumstances after the date of such statements.
- 3. Quick level set — We’re not going to talk about disaster recovery
- 5. Application Driver Primary Secondary Secondary Replication But what happens when you have data corruption in your primary or someone accidentally does something they shouldn’t have... ?
- 8. Data Recovery - That’s what we’re going to focus on today
- 9. Private DBaaS: On-Prem Public DBaaS: Fully Managed Built on the Same Code Base, Same API, Same Management UI MongoDB Offerings Hybrid DBaaS
- 10. The Future of Backup for Self Managing or On- Premises
- 11. On-Premises Backup Architecture Today Ops Manager OM01 OM02 OM03 OM04 OM05 OM06
- 12. On-Premises Backup Architecture Today Ops Manager OM01 OM02 OM03 OM04 OM05 OM06 Backup Agent Monitoring Agent Backup Agent Monitoring Agent Backup Agent Monitoring Agent Backup Agent Monitoring Agent Backup Agent Monitoring Agent Backup Agent Monitoring Agent OM Group
- 13. On-Premises Backup Architecture Today Ops Manager OM01 OM02 OM03 OM04 OM05 OM06 Backup Agent Monitoring Agent Backup Agent Monitoring Agent Backup Agent Monitoring Agent Backup Agent Monitoring Agent Backup Agent Monitoring Agent Backup Agent Monitoring Agent OM Group BackupLoadBalancer App Server for Backup App Server for Backup App Server for Backup
- 14. On-Premises Backup Architecture Today Ops Manager OM01 OM02 OM03 OM04 OM05 OM06 Backup Agent Monitoring Agent Backup Agent Monitoring Agent Backup Agent Monitoring Agent Backup Agent Monitoring Agent Backup Agent Monitoring Agent Backup Agent Monitoring Agent OM Group BackupLoadBalancer App Server for Backup App Server for Backup App Server for Backup Backup Daemon HeadDB Backup Daemon HeadDB Backup Daemon HeadDB
- 15. ... ... Blockstore06 Blockstore04 Blockstore02 On-Premises Backup Architecture Today Ops Manager OM01 OM02 OM03 OM04 OM05 OM06 Backup Agent Monitoring Agent Backup Agent Monitoring Agent Backup Agent Monitoring Agent Backup Agent Monitoring Agent Backup Agent Monitoring Agent Backup Agent Monitoring Agent OM Group BackupLoadBalancer App Server for Backup App Server for Backup App Server for Backup Backup Daemon HeadDB Backup Daemon HeadDB Backup Daemon HeadDB Blockstore01 Blockstore03 Blockstore05
- 16. ... ... ... ... Oplog04 Oplog02 Blockstore06 Blockstore04 Blockstore02 Oplog06 On-Premises Backup Architecture Today Ops Manager OM01 OM02 OM03 OM04 OM05 OM06 Backup Agent Monitoring Agent Backup Agent Monitoring Agent Backup Agent Monitoring Agent Backup Agent Monitoring Agent Backup Agent Monitoring Agent Backup Agent Monitoring Agent OM Group BackupLoadBalancer App Server for Backup App Server for Backup App Server for Backup Oplog01 Oplog03 Oplog05 Backup Daemon HeadDB Backup Daemon HeadDB Backup Daemon HeadDB Blockstore01 Blockstore03 Blockstore05
- 17. Backup Strategy ● Reduce the footprint and architectural complexities needed for backup ● Reduce cost ● Improve performance
- 19. What is it? ● Enhanced WiredTiger to take checkpoints of itself ● Checkpoints are then moved to long term storage ● HeadDB’s are completely eliminated — no more initial syncs! ○ Reduced storage requirements ○ Reduces architectural complexities ○ Reduces infrastructure required ● Consolidated agents, from three agents to one
- 20. ... ... ... ... Oplog04 Oplog02 Blockstore06 Blockstore04 Blockstore02 Oplog06 On-Premises Backup Architecture Today Ops Manager OM01 OM02 OM03 OM04 OM05 OM06 Backup Agent Monitoring Agent Backup Agent Monitoring Agent Backup Agent Monitoring Agent Backup Agent Monitoring Agent Backup Agent Monitoring Agent Backup Agent Monitoring Agent OM Group BackupLoadBalancer App Server for Backup App Server for Backup App Server for Backup Oplog01 Oplog03 Oplog05 Backup Daemon HeadDB Backup Daemon HeadDB Backup Daemon HeadDB Blockstore01 Blockstore03 Blockstore05
- 21. Oplog02 Blockstore New Backup Architecture Ops Manager OM01 OM02 OM03 OM04 OM05 OM06 MongoDB Agent MongoDB Agent MongoDB Agent MongoDB Agent MongoDB Agent MongoDB Agent OM Group BackupLoadBalancer App Server/Backup Daemon App Server/Backup Daemon Oplog01 S3 Snapshot Store
- 22. Backup Node Selection Order 1. Hidden Secondaries 2. A secondary we have already taken a snapshot from 3. Secondary closet to the time of the time of the snapshot 4. Any available secondary 5. Primary
- 23. Phase 1 Details • Backup agent required to exist on every node • OM 4.2 required for backing up 4.2 • Backwards compatible for 4.0 and below • Daemons are used for background tasks • Replica sets only, sharded clusters coming later • Advanced features coming later
- 24. S3 Oplog Blockstore New Backup Architecture — Future Ops Manager OM01 OM02 OM03 OM04 OM05 OM06 MongoDB Agent MongoDB Agent MongoDB Agent MongoDB Agent MongoDB Agent MongoDB Agent OM Group BackupLoadBalancer App Server for Backup App Server for Backup Oplog01 S3 Snapshot Store
- 25. Future Improvements ● Agents read/write directly to OpLog and snapshots store
- 26. S3 Oplog New Backup Architecture — Future Ops Manager OM01 OM02 OM03 OM04 OM05 OM06 MongoDB Agent MongoDB Agent MongoDB Agent MongoDB Agent MongoDB Agent MongoDB Agent OM Group S3 Snapshot Store
- 27. Future Improvements ● Agents read/write directly to OpLog and snapshots store ● Leave a checkpoint behind
- 28. What this means for you... ● Less people, time, and energy you have to spend to deliver a best-in-class data recovery solution for your on-premises and self-managed MongoDB deployments
- 29. Atlas
- 30. Private DBaaS: On-Prem Public DBaaS: Fully Managed Built on the Same Code Base, Same API, Same Management UI Atlas: Same Features as Ops Manager & Cloud Manager Hybrid DBaaS
- 31. The Latest MongoDB Features • MongoDB Atlas comes out-of-the-box with MongoDB 3.4, 3.6, 4.0 • Transactions (4.0) • Change Streams (3.6) • JSON Schema (3.6) • Expressive nested array updates (3.6) • Expressive joins: $lookup (3.6) • Graph queries (3.4) • Facets & expressive aggregations (3.4) • Minor updates and major upgrades without downtime ● Data Explorer ○ Inspect schema & index utilization ● Real-Time Performance Panel ○ Live telemetry: in-flight operations & resource consumption ● Performance Advisor ○ Always-on index recommendations
- 32. Multi-AZ Disaster Recovery by Default AZ IIAZ I AZ III
- 33. ...with Optional Multi-Region Fault Tolerance AZ IIAZ I AZ III Preferred Region AZ I AZ II AZ I AZ II Failover Region Failover Region
- 34. Atlas Backups: Data Recovery Roll back the clock when you run into issues triggered by user or application errors that are replicated from the primary to the rest of your cluster.
- 35. Point-in-Time Data Recovery • Lets you select a restore time based on your PIT window • Restores the closest snapshot and rolls ahead • Reduces the possibility of data loss
- 36. What About Small Disasters? • The application is working fine • But there is data missing or has been altered • No time to do a full restore
- 37. Queryable Backups • Ability to query your snapshots and restore data at the document level in minutes. • Reduces the operational overhead associated with: • Identifying whether data of interest has been altered • Pinpointing the best point in time to restore a database
- 38. db = source.locations db2 = destination.locations zips = db.zipcodes zips2 = db2.zipcodes def restore(): print "Finding Missing Data" query = {'state': ‘NY'} try: cursor = zips.find(query) except Exception as e: print "Unexpected error:", type(e), e for doc in cursor: zips2.insert(doc) Sample Queryable Script
- 39. Demo
- 41. Cloud Provider Snapshots (CPS) • Utilizes each providers native snapshot capabilities • Granular backup region selection • Satisfy data sovereignty requirements • Supports replica sets and sharded clusters • Pricing is based on snapshot size, not data size • Less expensive, starting at $0.08 per GB of snapshot size (varies per provider and region) • Now available on all cloud providers! • AWS and GCP snapshots are incremental Incremental snapshots for Atlas customers deploying on AWS 10 GB 15 GB (5 new) 20 GB (5 new)
- 42. Cloud Provider Snapshots (CPS) Updates — Continued • Bring your own keys • AWS KMS • Azure Key Vault • GCP KMS • Backup Policies • On-demand snapshots (quicksave) • Takes a snapshot immediately if there’s not already one in progress • API for pipeline integrations • M2/M5 Backups (Coming Soon!)
- 46. Whats Next?
- 47. Cloud Provider Snapshots - Future ● Point in Time Restores with customizable window ● Build a new cluster from a backup ● Queryable Backup with Selective restore
- 48. Pick the fully managed backup solution that’s right for you Within Atlas is a modern data recovery platform that enables you to choose how you want to backup your data, depending on your requirements. Continuous ● Point-in-time restore ○ All but eliminate the possibility of data loss ● Queryable snapshots ○ Rapidly restore granular data Cloud Provider Snapshots ● Localized backup ○ Satisfy data sovereignty requirements ● Bring your own keys ● Take snapshots on-demand ● The cost effective option