Optimizing Data Management for MongoDB
0 likes160 views
The document discusses the importance of optimizing data management for MongoDB, highlighting that data loss incidents can be costly for enterprises. It evaluates various backup options including mongodump, filesystem snapshots, and Ops Manager, noting their limitations in terms of recovery and resource intensity. Additionally, it introduces Imanis Data's architecture that offers advanced features for backup, recovery, and data management across multiple platforms.
Optimizing Data Management for MongoDB
- 1. Optimizing Data Management for MongoDB October 11, 2017
- 3. Why Bother With Backup and Test Data Mgmt? The average cost of a data loss incident is $900,000 90% of enterprises delay applications because of a lack of test data Source: EMC, Imanis Data
- 4. Data Replicas Don’t Prevent ALL Data Loss Human errors: dropping a collection Application corruption: incorrect updates to a collection Replicas go out of synch Primary Secondary Secondary
- 5. Current MongoDB Backup Options mongodump Filesystem/storage snapshots Ops Manager
- 6. MongoDB Backup/Recovery Options: mongodump Very resource intensive PROBLEM Not feasible for granular recovery PROBLEMPROBLEM Not incremental- forever PROBLEM
- 7. MongoDB Backup/Recovery Options: filesystem/storage snapshots Requires quiescing of MongoDB instance for consistent snapshots PROBLEM Requires periodic full backups to ensure faster recovery PROBLEM Cannot do point in time recovery or back up specific collections PROBLEM
- 8. MongoDB Backup/Recovery Options: Ops Manager Requires agents on production – scaling increases overhead PROBLEM PROBLEM Storage optimization is dependent on external media. Not data-aware PROBLEM Extremely difficult to restore to a different topology
- 9. Test Data Mgmt Eliminates Costly Application Delays Application teams wait for production data A company delayed 3-4 releases a year Cost the company $450K per year of companies delay application releases90% BUSINESS IMPACT CUSTOMER EXAMPLE PRIMARY CHALLENGE
- 10. Challenges With Typical Test Data Management Change Request - 1 week Provision Production Data - 1 week Create Test DB and Mask Data - 1 week Create Samples of Production Data – 2 days Push Production Data To Test – Hours Repeat Process – 3-4 weeks
- 11. The Evolution of Data Management THE NEXT 25 YEARS THE TRADITIONAL WORLD Data ManagementData Platforms
- 12. Imanis Data in Production Test Cluster Research Cluster Imanis Data GUI Hadoop/Spark Cluster Cassandra Cluster Vertica Cluster Couchbase Cluster Imanis Data Smart Storage Cluster MongoDB Cluster
- 13. The Imanis Data Architecture • Deep de-duplication and compression with app-aware architecture • Incremental-forever backup architecture • High availability via erasure coding in distributed cluster architecture Smart Storage Optimizer
- 14. The Imanis Data Architecture Native querying and analytics via active compute layer Unbounded scale with a Hadoop-native architecture Smart Storage Optimizer Active Compute Services Distributed File System
- 15. The Imanis Data Architecture • Google-like catalog shortens data recovery time • Automatic schema generation for mirroring and backups • Granular recovery at an object level • Recovery to multiple topologies • Native integration with LDAP and Kerberos for authentication • Role-based access control defines specific privileges • Stateless, consistent, irreversible, and one-way masks for PII data Smart Storage Optimizer Active Compute Services Distributed File System Metadata Catalog Data Orchestration ServicesSecurity Services
- 16. Smart Storage Optimizer The Imanis Data Architecture GUI CLI API Active Compute Services Distributed File System • ‘Single pane of glass’ for multiple use cases and data platforms • Agentless architecture minimizes management overhead • GUI, CLI, REST-based Talena API options Metadata Catalog Data Orchestration ServicesSecurity Services
- 17. Machine Intelligence: ThreatSense • Proactively identify anomalous data loss and ransomware to reduce downtime • Collects nearly 50 attributes to set baseline • Enables user input to optimize machine learning
- 18. Q&A