MongoDB Auto-Sharding at Mongo Seattle

MongoDBauto shardingAaron Stapleaaron@10gen.comMongo SeattleJuly 27, 2010

Why Scale Horizontally?Vertical scaling is expensiveHorizontal scaling is more incremental – works well in the cloudWill always be able to scale wider than higher

Distribution ModelsAd-hoc partitioningConsistent hashing (dynamo)Range based partitioning (BigTable/PNUTS)

Auto ShardingEach piece of data is exclusively controlled by a single node (shard)Each node (shard) has exclusive control over a well defined subset of the dataDatabase operations run against one shard when possible, multiple when necessaryAs system load changes, assignment of data to shards is rebalanced automatically

Mongo ShardingBasic goal is to make thismongodmongodmongod?client

Mongo ShardingLook like thismongodclient

Mongo ShardingMapping of documents to shards controlled by shard keyCan convert from single master to sharded cluster with 0 downtimeMost functionality of a single Mongo master is preservedFully consistent

Shard Key Examples{ user_id: 1 }{ state: 1 }{ lastname: 1, firstname: 1 }{ tag: 1, timestamp: -1 }{ _id: 1 }This is the defaultCareful when using ObjectId

Architecture Overview5500 <= user_id < +inf-inf <= user_id < 20002000 <= user_id < 5500mongodmongodmongodmongosclient

Query IShard key { user_id: 1 }db.users.find( { user_id: 5000 } )Query appropriate shard

Query IIShard key { user_id: 1 }db.users.find( { user_id: { $gt: 4000, $lt: 6000 } } )Query appropriate shard(s)

{ a : …, b : …, c : … } a is declared shard keyfind( { a : { $gt : 333, $lt : 400 } )

{ a : …, b : …, c : … } a is declared shard keyfind( { a : { $gt : 333, $lt : 2012 } )

Query IIIShard key { user_id: 1 }db.users.find( { hometown: ‘Seattle’ } )Query all shards

{ a : …, b : …, c : … } secondary query, secondary indexensureIndex({b:1})find( { b : 99 } )This case good when m is small (such as here), also when the queries are large tasks

Query IVShard key { user_id: 1 }db.users.find( { hometown: ‘Seattle’ } ).sort( { user_id: 1 } )Query all shards, in sequence

Query VShard key { user_id: 1 }db.users.find( { hometown: ‘Seattle’ } ).sort( { lastname: 1 } )Query all shards in parallel, perform merge sortSecondary index in { lastname: 1 } can be used

Map/ReduceMap/Reduce was designed for distributed systemsMap/Reduce jobs will run on all relevant shards in parallel, subject to query spec

Map/Reduce> m = function() { emit(this.user_id, 1); }> r = function(k,vals) { return 1; }> res = db.events.mapReduce(m, r, { query : {type:'sale'} });> db[res.result].find().limit(2){ "_id" : 8321073716060 , "value" : 1 }{ "_id" : 7921232311289 , "value" : 1 }

WritesInserts routed to appropriate shard (inserted doc must contain shard key)Removes call upon matching shardsUpdates call upon matching shardsWrites parallel if asynchronous, sequential if synchronousUpdates cannot modify the shard key

Choice of Shard KeyKey per document that is generally a component of your queriesOften you want a unique keyIf not, consider granularity of key and potentially add fieldsThe shard key is generally comprised of fields you would put in an index if operating on a single machineBut in a sharded configuration, the shard key will be indexed automatically

Shard Key Examples (again){ user_id: 1 }{ state: 1 }{ lastname: 1, firstname: 1 }{ tag: 1, timestamp: -1 }{ _id: 1 }This is the defaultCareful when using ObjectId

Bit.ly Example~50M users~10K concurrently using server at peak~12.5B shortens per month (1K/sec peak)History of all shortens per user stored in mongo

Bit.ly Example - Schema{ "_id" : "lthp", "domain" : null, "keyword" : null, "title" : "bit.ly, a simple urlshortener", "url" : "http://jay.bit.ly/", "labels" : [ {"text" : "lthp", "name" : "user_hash"}, {"text" : "BUFx", "name" : "global_hash"}, {"text" : "http://jay.bit.ly/", "name" : "url"}, {"text" : "bit.ly, a simple urlshortener", "name" : "title"}], "ts" : 1229375771, "global_hash" : "BUFx", "user" : "jay", "media_type" : null }

Bit.ly ExampleShard key { user: 1 }Indices{ _id: 1 }{ user: 1 }{ global_hash: 1 }{ user: 1, ts: -1 }Querydb.history.find( { user : user,labels.text : … } ).sort( { ts: -1 } )

BalancingThe whole point of autosharding is that mongo balances the shards for youThe balancing algorithm is complicated, basic idea is that this1000 <= user_id < +inf-inf <= user_id < 1000

BalancingBecomes this3000 <= user_id < +inf-inf <= user_id < 3000

BalancingCurrent balancing metric is data sizeFuture possibilities – cpu, disk utilizationThere is some flexibility built into the partitioning algorithm – so we aren’t thrashing data back and forth between shardsOnly move one ‘chunk’ of data at a time – a conservative choice that limits total overhead of balancing

ShardRegular mongodprocess(es), storing all documents for a given key rangeHandles all reads/writes for this key range as wellEach shard indexes the data contained within itCan be single mongod, master/slave, or replica set

Shard - ChunkIn a sharded cluster, shards partitioned by shard keyWithin a shard, chunks partitioned by shard keyA chunk is the smallest unit of data for balancingData moves between chunks at chunk granularityUpper limit on chunk size is 200MBSpecial case if shard key range is open ended

Shard - Replica SetsReplica sets provide data redundancy and auto failoverIn the case of sharding, this means redundancy and failover per shardAll typical replica set operations are possibleFor example, write with w=NReplica sets were specifically designed to work as shards

MongosSharding router – distributes reads/writes to sharded clusterClient interface is the same as a mongodCan have as many mongos instances as you wantCan run on app server machine to avoid extra network trafficMongos also initiates balancing operationsKeeps metadata per chunk in RAM – 1MB RAM per 1TB of user data in cluster

Config Server3 Config serversChanges are made with a 2 phase commitIf any of the 3 servers goes down, config data becomes read onlySharded cluster will remain online as long as 1 of the config servers is runningConfig metadata size estimate1MB metadata per 1TB data in cluster

LimitationsUnique index constraints not expressed by shard key are not enforced across shardsUpdates to a document’s shard key aren’t allowed (you can remove and reinsert, but it’s not atomic)Balancing metric is limited to # of chunks right now – but this will be enhancedRight now only one chunk moves in the cluster at a time – this means balancing can be slow, it’s a conservative choice we’ve made to keep the overhead of balancing low for now20 petabyte size limit

Start Up Config Servers$ mkdir -p ~/dbs/config$ ./mongod --dbpath ~/dbs/config --port 20000Repeat as necessary

Start Up Mongos$ ./mongos --port 30000 --configdb localhost:20000No dbpathRepeat as necessary

Start Up Shards$ mkdir -p ~/dbs/shard1 $ ./mongod --dbpath ~/dbs/shard1 --port 10000Repeat as necessary

Configure Shards$ ./mongo localhost:30000/admin> db.runCommand({addshard : "localhost:10000", allowLocal : true}){"added" : "localhost:10000", "ok" : true}Repeat as necessary

Shard Data> db.runCommand({"enablesharding" : "foo"})> db.runCommand({"shardcollection" : "foo.bar", "key" : {"_id" : 1}})Repeat as necessary

Production ConfigurationMultiple config serversMultiple mongos serversReplica sets for each shardUse getLastError/w correctly

Looking at config dataMongo shell connected to config server, config database> db.shards.find() { "_id" : "shard0", "host" : "localhost:10000" } { "_id" : "shard1", "host" : "localhost:10001" }

Looking at config data> db.databases.find() { "_id" : "admin", "partitioned" : false, "primary" : "config" } { "_id" : "foo", "partitioned" : false, "primary" : "shard1" } { "_id" : "x", "partitioned" : false, "primary" : "shard0” }{"_id" : "test", "partitioned" : true, "primary" : "shard0", "sharded" : { "test.foo" : { "key" : {"x" : 1}, "unique" : false } }}

Looking at config data> db.chunks.find() {"_id" : "test.foo-x_MinKey", "lastmod" : { "t" : 1276636243000, "i" : 1 }, "ns" : "test.foo", "min" : {"x" : { $minKey : 1 } }, "max" : {"x" : { $maxKey : 1 } }, "shard" : "shard0”}

Looking at config data> db.printShardingStatus() --- Sharding Status ---sharding version: { "_id" : 1, "version" : 3 } shards: { "_id" : "shard0", "host" : "localhost:10000" } { "_id" : "shard1", "host" : "localhost:10001" } databases: { "_id" : "admin", "partitioned" : false, "primary" : "config" } { "_id" : "foo", "partitioned" : false, "primary" : "shard1" } { "_id" : "x", "partitioned" : false, "primary" : "shard0" } { "_id" : "test", "partitioned" : true, "primary" : "shard0","sharded" : { "test.foo" : { "key" : { "x" : 1 }, "unique" : false } } }test.foo chunks: { "x" : { $minKey : 1 } } -->> { "x" : { $maxKey : 1 } } on : shard0 { "t" : 1276636243 …

Give it a Try!Download from mongodb.orgSharding production ready in 1.6, which is scheduled for release next weekFor now use 1.5 (unstable) to try sharding

MongoDB Auto-Sharding at Mongo Seattle

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to MongoDB Auto-Sharding at Mongo Seattle (20)

More from MongoDB (20)

Recently uploaded (20)

MongoDB Auto-Sharding at Mongo Seattle