How queries work with sharding
12 likes7,055 views
The document discusses how queries work in sharded MongoDB environments. It explains that MongoDB collections are partitioned into chunks based on a shard key, and each chunk is assigned to a particular shard. When a query is executed, the mongos process routes it to the correct shard(s) based on the shard key range in the query. Queries involving only the shard key are efficient, targeting specific shards. Queries on non-shard keys require scattering and gathering across all shards, but secondary indexes can help efficiency on each shard.
How queries work with sharding
- 1. MongoDB Sharding How queries work in sharded environments
- 2. One small server. We want more capacity. What to do?
- 3. TradiAonally, we would scale verAcally with a bigger box.
- 4. With sharding we instead scale horizontally to achieve the same computaAonal/storage/memory footprint from smaller servers. m=10
- 5. We will show the verAcally scale db and the horizontally scaled db side-‐by-‐side for comparison.
- 6. A sharded MongoDB collecAon has a shard key. The collecAon is parAAoned in an order-‐preserving manner using this key. In this example a is our shard key: { a : …, b : …, c : … } a is declared shard key for the collec0on
- 7. { a : …, b : …, c : … } a is shard key Metadata is maintained on chunks which are represented by shard key ranges. Each chunk is assigned to a parAcular shard. Range Shard a in [-‐∞,2000) 2 a in [2000,2100) 8 a in [2100,5500) 3 … … a in [88700, ∞) 0 When a chunk becomes too large, MongoDB automaAcally splits it, and the balancer will later migrate chunks as necessary.
- 8. { a : …, b : …, c : … } a is shard key find( { a : { $gt : 333, $lt : 400 } ) Range Shard a in [-‐∞,2000) 2 a in [2000,2100) 8 a in [2100,5500) 3 … … a in [88700, ∞) 0 The mongos process routes a query to the correct shard(s). For the query above, all data possibly relevant is on shard 2, so the query is sent to that node only, and processed there.
- 9. { a : …, b : …, c : … } a is declared shard key find( { a : { $gt : 333, $lt : 2012 } ) Range Shard a in [-‐∞,2000) 2 a in [2000,2100) 8 a in [2100,5500) 3 … … a in [88700, ∞) 0 SomeAmes a query range might span more than one shard, but very few in total. This is reasonably efficient.
- 10. { a : …, b : …, c : … } non-‐shard key query, no index find( { b : 99 } ) Range Shard a in [-‐∞,2000) 2 a in [2000,2100) 8 a in [2100,5500) 3 … … a in [88700, ∞) 0 Queries not involving the shard key will be sent to all shards as a “scader/gather” operaAon. This is someAmes ok. Here on both our tradiAonal machine and the shards, we do a table scan -‐-‐ equally expensive (roughly) on both.
- 11. { a : …, b : …, c : … } Sca8er / gather with secondary index ensureIndex({b:1}) find( { b : 99 } ) Range Shard a in [-‐∞,2000) 2 a in [2000,2100) 8 a in [2100,5500) 3 … … a in [88700, ∞) 0 Once again a query with a shard key results in a scader/gather operaAon. However at each shard, we can use the {b:1} index to make the operaAon efficient for that shard. We have a lidle extra overhead over the verAcal configuraAon for the communicaAons effort from mongos to each shard -‐-‐ not too bad if number of shards is small (10) but quite substanAal for say, a 1000 shard system.
- 12. { a : …, b : …, c : … } Non-‐shard key query, secondary index find( { b : 99, a : 100 } ) Range Shard a in [-‐∞,2000) 2 a in [2000,2100) 8 a in [2100,5500) 3 … … a in [88700, ∞) 0 The a term involves the shard key and allows mongos to intelligently route the query to shard 2. Once the query reaches shard 2, the { b : 1 } index can be used to efficiently process the query.
- 13. When sorAng is specified, the relevant shards sort locally, and then mongos merges the results. Thus the mongos resource usage is not terribly high. client Adam Bob David Julie Sue Time Zack mongos Bob David Sue Adam Julie Tim Zack
- 14. When using replicaAon (typically a replica set), we simply have more than one node per shard. (arrows below indicate replica0on, tradi0onal vs. sharded environments)