MongoDB at Scale
Download as PPTX, PDF18 likes7,274 views
This document discusses how to achieve scale with MongoDB. It covers optimization tips like schema design, indexing, and monitoring. Vertical scaling involves upgrading hardware like RAM and SSDs. Horizontal scaling involves adding shards to distribute load. The document also discusses how MongoDB scales for large customers through examples of deployments handling high throughput and large datasets.
![Document Model
• Matches Application
Objects
• Flexible
• High performance
{ "customer_id" : 123,
"first_name" : ”John",
"last_name" : "Smith",
"address" : {
"street": "123 Main Street",
"city": "Houston",
"state": "TX",
"zip_code": "77027"
}
policies: [ {
policy_number : 13,
description: “short term”,
deductible: 500
},
{ policy_number : 14,
description: “dental”,
visits: […]
} ]
}](https://image.slidesharecdn.com/mongodb-at-scale-v031-150115131940-conversion-gate01/85/MongoDB-at-Scale-6-320.jpg)
![mongod log files
Sun Jun 29 06:35:37.646 [conn2]
query test.docs query: {
parent.company: "22794",
parent.employeeId: "83881" }
ntoreturn:1 ntoskip:0
nscanned:806381 keyUpdates:0
numYields: 5 locks(micros)
r:2145254 nreturned:0 reslen:20
1156ms](https://image.slidesharecdn.com/mongodb-at-scale-v031-150115131940-conversion-gate01/85/MongoDB-at-Scale-18-320.jpg)
MongoDB at Scale
- 2. Achieving Scale with MongoDB Thomas Boyd Manager, SolutionsArchitecture, MongoDB
- 3. MongoDB Doesn’t Scale!! 0 5,000 10,000 15,000 20,000 25,000 30,000 1 2 3 4 5 6 7 8 Operations/Second Number of Nodes MongoDB Cluster Throughput
- 4. Agenda • Optimization Tips – Schema Design – Indexes – Monitoring • Vertical Scaling • Horizontal Scaling • Scaling your Operations Team • Customer @ Scale on MongoDB
- 5. Optimization Tips: Schema Design
- 6. Document Model • Matches Application Objects • Flexible • High performance { "customer_id" : 123, "first_name" : ”John", "last_name" : "Smith", "address" : { "street": "123 Main Street", "city": "Houston", "state": "TX", "zip_code": "77027" } policies: [ { policy_number : 13, description: “short term”, deductible: 500 }, { policy_number : 14, description: “dental”, visits: […] } ] }
- 7. The Importance of Schema Design • Very different from RDBMS schema design • MongoDB Schema: – denormalize the data – create a (potentially complex) schema with prior knowledge of your actual (not just predicted) query patterns – write simple queries
- 8. Real World Example Product catalog for retailer selling in 20 countries { _id: 375, en_US: { name: …, description: …, <etc…> }, en_GB: { name: …, description: …, <etc…> }, fr_FR: { name: …, description: …, <etc…> }, fr_CA: { name: …, description: …, <etc…> }, de_DE: …, <… and so on for other locales …> }
- 9. Not a Good Match for Access Pattern Actual application queries: db.catalog.find( { _id: 375 }, { en_US: true } ); db.catalog.find( { _id: 375 }, { fr_FR: true } ); db.catalog.find( { _id: 375 }, { de_DE: true } ); … and so forth for other locales
- 10. Inefficient use of resources Data in RED are being used. Data in BLUE take up memory but are not in demand. { _id: 375, en_US: { name: …, description: …, <etc…> }, en_GB: { name: …, description: …, <etc…> }, fr_FR: { name: …, description: …, <etc…> }, fr_CA: { name: …, description: …, <etc…> }, de_DE: …, de_CH: …, <… and so on for other locales …> } { _id: 42, en_US: { name: …, description: …, <etc…> }, en_GB: { name: …, description: …, <etc…> }, fr_FR: { name: …, description: …, <etc…> }, fr_CA: { name: …, description: …, <etc…> }, de_DE: …, de_CH: …, <… and so on for other locales …> }
- 11. Consequences of Schema Redesign • Queries induced minimal memory overhead • 20x as many products fit in RAM at once • Disk IO utilization reduced • Application latency reduced { _id: "375-en_GB", name: …, description: …, <… the rest of the document …> }
- 12. Schema Design Patterns • Pattern: pre-computing interesting quantities, ideally with each write operation • Pattern: putting unrelated items in different collections to take advantage of indexing • Anti-pattern: appending to arrays ad infinitum • Anti-pattern: importing relational schemas directly into MongoDB
- 13. TAKE THE TIME TO UNDERSTAND YOUR APPLICATION
- 14. Schema Design Resources • Blog series, "6 rules of thumb" – Part 1: http://goo.gl/TFJ3dr – Part 2: http://goo.gl/qTdGhP – Part 3: http://goo.gl/JFO1pI • Webinars, training, consulting, etc…
- 16. B-Tree Indexes • Tree-structured references to your documents • Single biggest tunable performance factor • Indexing and schema design go hand in hand
- 17. Indexing Mistakes and Their Fixes • Failing to build necessary indexes – Run .explain(), examine slow query log, mtools, system.profile collection • Building unnecessary indexes – Talk to your application developers about usage • Running ad-hoc queries in production – Use a staging environment, use secondaries
- 18. mongod log files Sun Jun 29 06:35:37.646 [conn2] query test.docs query: { parent.company: "22794", parent.employeeId: "83881" } ntoreturn:1 ntoskip:0 nscanned:806381 keyUpdates:0 numYields: 5 locks(micros) r:2145254 nreturned:0 reslen:20 1156ms
- 19. mtools • http://github.com/rueckstiess/mtools • log file analysis for poorly performing queries – Show me queries that took more than 1000 ms from 6 am to 6 pm: – mlogfilter mongodb.log --from 06:00 --to 18:00 --slow 1000 > mongodb-filtered.log
- 20. Indexing Strategies • Create indexes that support your queries! • Create highly selective indexes • Eliminate duplicate indexes with compound indexes – db.collection.ensureIndex({A:1, B:1, C:1}) – allows queries using leftmost prefix • Order index columns to support scans & sorts • Create indexes that support covered queries • Prevent collection scans in pre-production environments db.getSiblingDB("admin").runCommand( { setParameter: 1, notablescan: 1 } )
- 22. JUST DO IT NOW IN PRE-PROD
- 23. MongoDB Management Services (MMS) Backup Monitoring Automation
- 24. MMS: Database and Hardware Metrics
- 26. Cloud Version of MMS 1. Go to http://mms.mongodb.com 2. Create an account 3. Install one agent in your datacenter 4. Add hosts from the web interface 5. Enjoy!
- 27. Vertical Scaling
- 28. Factors: – RAM – Disk – CPU – Network We are Here to Pump you Up Primary Secondary Secondary Replica Set Primary Secondary Secondary Replica Set
- 29. Working Set Exceeds Physical Memory
- 30. Real world Example • Status changes for entities in the business • State changes happen in batches and are fully random – sometimes 10% of entities get updated – sometimes 100% get updated
- 31. Initial Architecture Sharded Cluster, 4 shards backed by spinning disk Application / mongos mongod
- 32. Horizontal Scaling Rapidly growing business means more shards Application / mongos …16 more shards… mongod
- 33. Vertical Scaling Scaling random IOPS with SSDs Application / mongos mongod SSD
- 34. Before you add hardware.... • Make sure you are solving the right scaling problem • Remedy schema and index problems first – schema and index problems can look like hardware problems • Tune the Operating System – ulimits, swap, NUMA, NOOP scheduler with hypervisors • Tune the IO subsystem – ext4 or XFS vs SAN, RAID10, readahead, noatime • See MongoDB "production notes" page • Heed logfile startup warnings
- 36. Why Shard? • Space • Throughput • Latency • (Manageability of individual nodes) – restore, compaction, indexing, etc…
- 37. Sharding Overview Primary Secondary Secondary Shard 1 Primary Secondary Secondary Shard 2 Primary Secondary Secondary Shard 3 Primary Secondary Secondary Shard N … Query Router Query Router Query Router …… Driver Application
- 38. Range Sharding mongod Read/Write Scalability Key Range 0..100
- 39. Range Sharding Read/Write Scalability mongod mongod Key Range 0..50 Key Range 51..100
- 40. Sharding mongod mongod mongod mongod Key Range 0..25 Key Range 26..50 Key Range 51..75 Key Range 76.. 100 Read/Write Scalability
- 41. Shard Key Characteristics • A good shard key has: – sufficient cardinality – distributed writes – targeted reads ("query isolation") • Shard key should be in every query if possible – scatter gather otherwise • Choosing a good shard key is important! – affects performance and scalability – changing it later is expensive
- 42. Beware of Ascending Shard Keys • Monotonically increasing shard key values cause "hot spots" on inserts • Examples: timestamps, _id Shard 1 mongos Shard 2 Shard 3 Shard N [ ISODate(…), $maxKey )
- 43. Beware of Scatter-Gather Queries • Extra network traffic • Extra work on each node • Sorts in mongos Primary Secondary Secondary Primary Secondary Secondary Primary Secondary Secondary Primary Secondary Secondary … Query Router Query Router Query Router …… Driver Application
- 44. Advanced Sharding Options • Hash-based Sharding • Tag-Aware Sharding mongod mongod mongod mongod Shard Tag Start End Winter 23 Dec 21 Mar Spring 22 Mar 21 Jun Summer 21 Jun 23 Sep Fall 24 Sep 22 Dec Spring Summer FallWinter
- 45. I FORGOT THE SECTION ON REPLICA SETS !?
- 46. Scaling your Operations Team
- 47. How MMS helps you Scale EasilyMeet SLAs Best Practices, Automated Cut Management Overhead
- 48. Without MMS Example Deployment – 12 Servers Install, Configure 150+ steps …Error handling, throttling, alerts Scale out, move servers, resize oplog, etc. 10-180+ steps Upgrades, downgrades 100+ steps
- 49. With MMS
- 50. Common Tasks, Performed in Minutes • Deploy – any size, most topologies • Upgrade/Downgrade – with no downtime • Scale – add/remove shards or replicas, with no downtime • Resize Oplog – with no downtime • Specify users, roles, custom roles • Provision AWS instances and optimize for MongoDB
- 51. Customers @ Scale on MongoDB
- 52. MonoDB at Scale 250M Ticks/Sec 300K+ Ops/Sec 500K+ Ops/SecFed Agency Performance 1,400 Servers 1,000+ Servers 250+ Servers Entertainment Co. Cluster Petabytes 10s of billions of objects 13B documents Data Asian Internet Co.
- 53. Foursquare Stats • 50M users. • 1.7M merchants using the platform for marketing • Operations Per Second: 300,000 • Documents: 5.5B • 11 MongoDB clusters – 8 are sharded • Largest cluster has 15 shards (check ins) – Sharded on user id
- 54. MongoDB DOES Scale!! 0 5,000 10,000 15,000 20,000 25,000 30,000 1 2 3 4 5 6 7 8 Operations/Second Number of Nodes MongoDB Cluster Throughput
Editor's Notes
- #48: MMS can do a lot for [ops teams]. Best Practices, Automated. MMS takes best practices for running MongoDB and automates them. So you run ops the way MongoDB engineers would do it. This not only makes it more fool-proof, but it also helps you… Cut Management Overhead. No custom scripting or special setup needed. You can spend less time running and managing manual tasks because MMS takes care of a lot of the work for you, letting you focus on other tasks. Meet SLAs. Automating critical management tasks makes it easier to meet uptime SLAs. This includes managing failover as well as doing rolling upgrades with no downtime. Scale Easily. Provision new nodes and systems with a single click.
- #49: It is, of course, possible to do these things without MMS. But it takes work. Typically manual work, or custom scripting. In either case, these things take time, require you to check for mistakes and are more prone to having things go wrong.
- #53: More info: http://www.mongodb.com/mongodb-scale