![Documents
• Common MongoDB BSON Types
• ObjectId
• String
• Integer
• Double
• NumberLong
• Date
• Boolean
• Null
• Array
{
"_id" : ObjectId("570680d891442f6efaff2005"),
"n" : "John Doe",
"a" : 45,
"h" : 5.9,
"w" : NumberLong(165),
"u" : ISODate("2016-01-07T15:46:32.085Z"),
"ac" : true,
"nn" : null,
"z" : [
"10013",
"10018"
]
}](https://image.slidesharecdn.com/mongodbchunks-180421131626/85/MongoDB-Chunks-Distribution-Splitting-and-Merging-4-320.jpg)
![Use Cases For Splitting (Case 1)
Profiler Example :
db.system.profile.aggregate([
{$match: { $and: [ {op:"update"}, {ns : "mydb.mycoll"} ] }}, {$group: { "_id":"$query.uuid",
count:{$sum:1}}},
{$sort: {"count": -1}}, {$limit : 5 } ]);
{
"result" : [
{
"_id" : "00005cf6-1217-4414-935b-
cf1bde09cc77",
"count" : 28672
},
..........
],
"ok" : 1
}](https://image.slidesharecdn.com/mongodbchunks-180421131626/85/MongoDB-Chunks-Distribution-Splitting-and-Merging-8-320.jpg)
![Moving the Chunks
db.runCommand({
"moveChunk" : "mydb.mycoll",
"bounds" : [
{
"uuid" : "7fe55742-7879-44bf-9a00-462a0284c982"
},
{
"uuid" : "ffffe5bc-d04d-4fcf-b8e2-283fa5998079"
}
],
"to" : "s2",
"_secondaryThrottle" : true
});
Example Command:](https://image.slidesharecdn.com/mongodbchunks-180421131626/85/MongoDB-Chunks-Distribution-Splitting-and-Merging-10-320.jpg)
MongoDB Chunks - Distribution, Splitting, and Merging
- 1. MongoDB Chunks Jason Terpko MongoDB Chunks – Distribution, Splitting, and Merging NoSQL DBA, Rackspace/ObjectRocket www.linkedin.com/in/jterpko, [email protected]
- 2. My Story • Started out in relational databases in public education then financial services • Next came online media distribution combined with a paywall • For analytics, started working with columnar databases and engines with compression • Made the switch to NoSQL at ObjectRocket by Rackspace
- 3. Overview • MongoDB and Sharding • What is a chunk in MongoDB? • Chunk Distribution and Scaling MongoDB • Use Cases For Splitting • Use Cases For Merging • Reconsidering Your Shard Key
- 4. Documents • Common MongoDB BSON Types • ObjectId • String • Integer • Double • NumberLong • Date • Boolean • Null • Array { "_id" : ObjectId("570680d891442f6efaff2005"), "n" : "John Doe", "a" : 45, "h" : 5.9, "w" : NumberLong(165), "u" : ISODate("2016-01-07T15:46:32.085Z"), "ac" : true, "nn" : null, "z" : [ "10013", "10018" ] }
- 6. What is a Chunk? config.chunks { "_id" : "mydb.mycoll-uuid_"00005cf6-1217-4414-935b- cf1bde09cc77"", "lastmod" : Timestamp(1, 5), "lastmodEpoch" : ObjectId("570733145f2bf94777a62155"), "ns" : "mydb.mycoll", "min" : { "uuid" : "00005cf6-1217-4414-935b-cf1bde09cc77" }, "max" : { "uuid" : "7fe55637-74c0-4e51-8eed-ab6b411d2b6e" }, "shard" : ”s1" }
- 7. Use Cases For Splitting (Case 1) s1 s2 80% 20% Unbalanced Operations
- 8. Use Cases For Splitting (Case 1) Profiler Example : db.system.profile.aggregate([ {$match: { $and: [ {op:"update"}, {ns : "mydb.mycoll"} ] }}, {$group: { "_id":"$query.uuid", count:{$sum:1}}}, {$sort: {"count": -1}}, {$limit : 5 } ]); { "result" : [ { "_id" : "00005cf6-1217-4414-935b- cf1bde09cc77", "count" : 28672 }, .......... ], "ok" : 1 }
- 9. Use Cases For Splitting (Case 1) { "_id" : "mydb.mycoll-uuid_"00005cf6-1217-4414-935b-cf1bde09cc77"", "lastmod" : Timestamp(1, 5), "lastmodEpoch" : ObjectId("570733145f2bf94777a62155"), "ns" : "mydb.mycoll", "min" : { "uuid" : "00005cf6-1217-4414-935b-cf1bde09cc77" }, "max" : { "uuid" : "7fe55637-74c0-4e51-8eed-ab6b411d2b6e" }, "shard" : ”s1" } { “uuid” : “7800e10e-273d-4186-bca7-9e3f6647d33a”, … { “uuid” : “7801a55e-e326-4cdb-a5fb-a5d133016f13” , … { “uuid” : “7801c3d5-8506-4715-934b-ad66904ae01e” , … { “uuid” : “7801ff57-50b7-4505-9557-06058cc8ff80” , … sh.splitAt( ) Unbalanced Operations
- 10. Moving the Chunks db.runCommand({ "moveChunk" : "mydb.mycoll", "bounds" : [ { "uuid" : "7fe55742-7879-44bf-9a00-462a0284c982" }, { "uuid" : "ffffe5bc-d04d-4fcf-b8e2-283fa5998079" } ], "to" : "s2", "_secondaryThrottle" : true }); Example Command:
- 11. Use Cases For Splitting (Case 2) s1 s2 s1 s2 Unbalanced Resources
- 12. Use Cases For Splitting (Case 2) Unbalanced Resources – Jumbo Chunks • Requirements • Larger than db.settings.find({"_id" : "chunksize"}) • Or contains 250,000 or more documents • Causes • Importing of data • Config metadata lock • Compound Shard Key • Identifying • sh.status(true) • moveChunk() • count() or Aggregation Pipeline • ObjectRocket Utility : ChunkHunter.py* • Rectifying • sh.splitAt() and sh.splitFind() • ObjectRocket Utilities : ChunkManager.py* • Data reduction • Re-sharding the collection *https://github.com/objectrocket/Utils
- 13. Splitting : Identifying and Rectifying Aggregation Example: ObjectRocket Utils : ChunkManager.py ChunkHunter.py ChunkSplitter.pyCn t exit db.mycoll.aggregate([{$group: {"_id": {"uuid": "$uuid"}, "count": {$sum: 1}}},{$sort: {"count": -1}},{$lim db.adminCommand({ dataSize: "mydb.mycoll", keyPattern: { "uuid" : 1 }, min: { "uuid" : "7fe55637-74c0-4e51-8eed-ab6b411d2b6e" }, max: { "uuid" : "7fe55742-7879-44bf-9a00-462a0284c982" } }) Data Size:
- 14. Use Cases For Splitting (Case 3) s1 s2 100% New and Small Collections
- 15. Use Cases For Splitting (Case 3) New and Small Collections • Considerations – Workload – Shard Key – Shard Count – Current Collection Size – Expected Collection Size • Command db.runCommand( { shardCollection: "mydb.mycoll", key: { ”uuid": "hashed" }, numInitialChunks : 1024 } ) • Calculating The Value (size_in_mb / chunk_size) * 2
- 16. Use Cases For Merging s1 s2 80% 20% s1 s2 Unbalanced Operations Unbalanced Resources Empty Chunks { "uuid" : "00005cf6-....." } -->> { "uuid" : "7fe55637-....." } on : s1 100K Docs @ 45MB { "uuid" : "7fe55637-....." } -->> { "uuid" : "7fe55742-....." } on : s2 0 Docs @ 0MB
- 17. Use Cases For Merging (Deletes) Unexpected remove() Operations insert() & split insert() & split insert() -> split -> remove() insert() -> split -> remove() s1 s2 s3 s4
- 18. Merging Process How we have resolved this with JavaScript in past: 1. Check Balancer State 2. Read in empty chunks from ChunkHunter.py results collection 3. Locate adjacent chunk 4. If empty and adjacent chunks reside on the same shard, merge 5. Else move chunk to the shard with the adjacent chunk, then merge What have we learned from the current process?
- 19. Use Cases For Merging (TTL) TTL : db.mycoll.ensureIndex( { "ca": 1 }, { expireAfterSeconds: 5227200 } ) splitVector & split moveChunk() s1 s2 s3 s4
- 20. Chunk Size and Splits How frequently are chunks being split? • Global Change • split & splitVector • Decreasing the size • Increased the size
- 21. Review Your Shard Key How frequently does this occur? • Is this a re-occurring problem? • What impact does it have to your business? • Re-analyzing your structure, workload, and access patterns • What method will you use to re-shard a sharded collection?
- 23. Questions? • MongoDB • Sharding • Chunks • Splitting Chunks • Merging Chunks • ObjectRocket • Rackspace