Webinar: Capacity Planning
Download as PPTX, PDF•2 likes•1,549 views
The document discusses capacity planning for deploying MongoDB, emphasizing its importance, requirements, and the consequences of inadequate planning. It covers key factors such as hardware resource usage, IOPS examples, and measurement methods, along with the need for continuous monitoring and evaluation. The document also highlights common questions to consider for effective capacity planning.
Webinar: Capacity Planning
- 1. Server Engineer Shaun Verch Capacity Planning: Deploying MongoDB
- 2. Capacity Planning • Why is it important? • What is it? • When is it important? • How is it actually done?
- 3. Why?
- 4. • What are the consequences of not planning? Why does it matter?
- 5. What?
- 6. What is Capacity Planning? Requirements Resources
- 7. • Availability • Throughput • Responsiveness Requirements
- 8. • Availability • Throughput • Responsivenes s Requirements to Hardware
- 9. Resource Usage • Storage – IOPS – Size – Data & Loading Patterns • Memory – Working Set • CPU – Speed – Cores • Network – Latency – Throughput
- 10. Storage • Active • Archival • Loading Patterns • Integration (BI/DW)
- 11. Storage • Active • Archival • Loading Patterns • Integration (BI/DW) Example IOPS
- 12. Example IOPS 7,200 rpm SATA ~ 75-100 IOPS 15,000 rpm SAS ~ 175-210 IOPS Amazon EBS/Provisioned ~ 100 IOPS "up to" 2,000 IOPS Amazon SSD 9,000 – 120,000 IOPS Storage Capability
- 13. Intel X25-E (SLC) ~ 5,000 IOPS Fusion IO ~ 135,000 IOPS Violin Memory 6000 ~ 1,000,000 IOPS Example IOPS 7,200 rpm SATA ~ 75-100 IOPS 15,000 rpm SAS ~ 175-210 IOPS Amazon EBS/Provisioned ~ 100 IOPS "up to" 2,000 IOPS Amazon SSD 9,000 – 120,000 IOPS Storage Capability
- 14. Intel X25-E (SLC) ~ 5,000 IOPS Fusion IO ~ 135,000 IOPS Violin Memory 6000 ~ 1,000,000 IOPS Cost of IOPS 7,200 rpm SATA ~ 75-100 IOPS 15,000 rpm SAS ~ 175-210 IOPS Amazon EBS/Provisioned ~ 100 IOPS "up to" 2,000 IOPS Amazon SSD 9,000 – 120,000 IOPS Storage Costs
- 15. Memory • Working Set – Active Data in Memory – Measured Over Periods
- 18. Working Set Number of distinct pages accessed per unit of time
- 19. Working Set Number of distinct pages accessed per second
- 20. Working Set 4 distinct pages per second
- 21. Working Set 4 distinct pages per second
- 22. Working Set 4 distinct pages per second Worst case 4 disk accesses
- 23. Working Set 6 distinct pages per second
- 24. Working Set 6 distinct pages per second
- 25. Working Set 6 distinct pages per second
- 26. Working Set 6 distinct pages per second Worst case disk access on every op
- 28. CPU • Non-indexed Data • Sorting • Aggregation – Map/Reduce – Framework • Data – Fields – Nesting – Arrays/Embedded-Docs
- 29. Network • Latency – WriteConcern – ReadPreference – Batching • Throughput – Update/Write Patterns – Reads/Queries
- 30. What is failure? • We have failed at Capacity Planning when our resources don’t meet our requirements • Because our requirements can have many dimensions, we may exceed our requirements in one characteristic but not meet them in another • This means that we can spend many $$$ and still fail!
- 31. What about Legacy Hardware? • Let’s hope whatever worked for this legacy technology also works for MongoDB • Same principles of Capacity Planning still apply
- 32. When?
- 33. • Before it's too late! • When? Capacity Planning: When Start Launch Version 2
- 34. Capacity Planning is Measurement Measuring early gives you a comparison point for when you need to do it again
- 35. Velocity of Change • Limitations -> takes time – Data Movement – Allocation/Provisioning (servers/mem/disk) • Improvement – Limit Size of Change (if you can) – Increase Frequency – MEASURE its effect – Practice
- 36. Repeat (continuously) • Repeat Testing • Repeat Evaluations • Repeat Deployment
- 37. How?
- 38. Monitoring Monitoring Storage Memory CPU Network Application Metrics
- 39. Tools • MMS (MongoDB Monitoring Service) • MongoDB: mongotop, mongostat • Linux: iostat, vmstat, sar, etc • Windows: Perfmon Measure realistic loads (generated by Load testing)
- 40. Models • Load/Users – Response Time/TTFB • System Performance – Peak Usage – Min Usage
- 41. Starter Questions • What is the working set? – How does that equate to memory – How much disk access will that require • How efficient are the queries? • What is the rate of data change? • How big are the highs and lows?
- 42. Deployment Types All of these use the same resources: • Single Instance • Multiple Instances (Replica Set) • Cluster (Sharding) • Data Centers
- 43. Questions?
- 44. Server Engineer, MongoDB Shaun Verch Thank You
Editor's Notes
- #5: Doesn’t meet your requirements
- #6: Understand trade-offs!
- #7: 7: Why, What, When, Where, How, WhoSkipping Who is you – that's why you're here, right?Where – hopefully on the same systems you'll be running in production, How -- bonusShow evidenceCapacity planning prerequisitesWhy do we need to do this?Epic fail – what's the consequence of not doing this?Thankless job – if you get it right no one notices. But if you get it wrong...Failure to project performance drop-off as the amount of data increases
- #8: Once we launch, we don't want to have avoidable down time due to poorly selected HWAs our success grows we want to stay in front of the demand curveWe want to meet business' and users' expectationsWe want to keep our jobs and get big raises! ;)
- #17: Initialize -> ElectionPrimary + data replication from primary to secondary
- #18: Break into separate slides - Show actual examples (possibly from mms) illustrating these points
- #39: Note, change the color to match the actual things.
- #40: top three – all add to memory what happens in RAM...if you have 64GB – 20GB sorting aggregation and connectionshow that each using more decreases availability for others2.4 will add a way to estimate "working set"
- #41: top three – all add to memory what happens in RAM...if you have 64GB – 20GB sorting aggregation and connectionshow that each using more decreases availability for others2.4 will add a way to estimate "working set"
- #43: Take out “Documents (and Collections)”
- #53: PriorityFloating point number between 0..1000Highest member that is up to date wins Up to date == within 10 seconds of primaryIf a higher priority member catches up, it will force election and win Slave DelayLags behind master by configurable time delay Automatically hidden from clientsProtects against operator errorsFat fingeringApplication corrupts data
- #61: PriorityFloating point number between 0..1000Highest member that is up to date wins Up to date == within 10 seconds of primaryIf a higher priority member catches up, it will force election and win Slave DelayLags behind master by configurable time delay Automatically hidden from clientsProtects against operator errorsFat fingeringApplication corrupts data