Download to read offline
![> conf = { // 5 data nodes
_id : "mySet",
members : [
{_id : 0, host : "A”, priority : 3},
{_id : 1, host : "B", priority : 2},
{_id : 2, host : "C”},
{_id : 3, host : "D", hidden : true},
{_id : 4, host : "E", hidden : true, slaveDelay : 3600}
]
}
> rs.initiate(conf)
Configuration Options](https://image.slidesharecdn.com/2014-05-07-fr-adddevseries-session6-deployingyourapplication-2-140516054744-phpapp01/75/2014-05-07-fr-add-dev-series-session-6-deploying-your-application-2-17-2048.jpg)
![{
_id : "mySet",
members : [
{_id : 0, host : "A", tags : {"dc": "ny"}},
{_id : 1, host : "B", tags : {"dc": "ny"}},
{_id : 2, host : "C", tags : {"dc": "sf"}},
{_id : 3, host : "D", tags : {"dc": "sf"}},
{_id : 4, host : "E", tags : {"dc": "cloud"}}],
settings : {
getLastErrorModes : {
allDCs : {"dc" : 3},
someDCs : {"dc" : 2}} }
}
> db.blogs.insert({...})
> db.runCommand({getLastError : 1, w : "someDCs"})
Tagging Example](https://image.slidesharecdn.com/2014-05-07-fr-adddevseries-session6-deployingyourapplication-2-140516054744-phpapp01/75/2014-05-07-fr-add-dev-series-session-6-deploying-your-application-2-23-2048.jpg)
![db.runCommand({ listshards:1 })
{ "shards" :
[{"_id”:"shard0000”,"host”:”<hostname>:27018”} ],
"ok" : 1
}
Verify that the shard was added](https://image.slidesharecdn.com/2014-05-07-fr-adddevseries-session6-deployingyourapplication-2-140516054744-phpapp01/75/2014-05-07-fr-add-dev-series-session-6-deploying-your-application-2-53-2048.jpg)
Uploaded byMongoDB
2014 05-07-fr - add dev series - session 6 - deploying your application-2
The document discusses MongoDB replication and sharding. Replication uses replica sets for high availability and disaster recovery. Sharding partitions data across multiple servers (shards) to improve scalability. The key points covered include: - Replication maintains copies of data on multiple servers for redundancy and high availability. It uses replica sets and elections for failover. - Sharding partitions data by a shard key across multiple mongod instances (shards) to scale reads and writes. It requires config servers to store metadata and mongos instances as query routers. - Write concerns allow controlling acknowledgments and replication of write operations. Tag-aware sharding allows controlling data distribution across shards.
More Related Content
Similar to 2014 05-07-fr - add dev series - session 6 - deploying your application-2
![MongoDB .local San Francisco 2020: Powering the new age data demands [Infosys]](https://cdn.slidesharecdn.com/ss_thumbnails/315pminfosysfinalsfoversionvocalpart1-200120221508-thumbnail.jpg?width=640&height=640&fit=bounds)
2014 05-07-fr - add dev series - session 6 - deploying your application-2
- 1. Tugdual Grall (@tgrall) AlainHélaïli (@AlainHelaili) #MongoDBBasics @MongoDB Construire une application avec MongoDB Déploiement de l’application
- 2. 2 • Résumé desépisodes précédents • Replication • Sharding Agenda
- 3. 3 • Virtual GeniusBar – Utilisez la fenêtre de chat – Tug & Alain dispo pendant, et après… • MUGs à Paris, Toulouse, Bordeaux, Rennes, Lyon • Groupes LinkedIn « MongoDB France » et « MongoDB » sur Viadeo Q & A @tgrall, [email protected] - @AlainHelaili, [email protected]
- 4.
- 5. 5 • Agrégation dedonnées… – Map Reduce – Hadoop – Rapports Pré-Agrégés – Aggregation Framework • Tuning avec Explain • Calcul à la volée ou calcul/stocke • Geospatial • Text Search Résumé
- 6.
- 7. Why Replication? • Howmany have faced node failures? • How many have been woken up from sleep to do a fail-over(s)? • How many have experienced issues due to network latency? • Different uses for data – Normal processing – Simple analytics
- 8. Why Replication? • Replicationis designed for – High Availability (HA) – Disaster Recovery (DR) • Not designed for scaling reads – You can but there are drawbacks: eventual consistency, etc. – Use sharding for scaling!
- 9. Replica Set –Creation
- 10. Replica Set –Initialize
- 11. Replica Set –Failure
- 12. Replica Set –Failover
- 13. Replica Set –Recovery
- 14. Replica Set –Recovered
- 15. Replica Set Roles& Configuration
- 16. Replica Set Roles Examplewith 2 data nodes + 1 arbiter
- 17. > conf ={ // 5 data nodes _id : "mySet", members : [ {_id : 0, host : "A”, priority : 3}, {_id : 1, host : "B", priority : 2}, {_id : 2, host : "C”}, {_id : 3, host : "D", hidden : true}, {_id : 4, host : "E", hidden : true, slaveDelay : 3600} ] } > rs.initiate(conf) Configuration Options
- 18.
- 19.
- 20. Delayed / EventualConsistency
- 21. Write Concerns • Networkacknowledgement (w = 0) • Wait for return info/error (w = 1) • Wait for journal sync (j = 1) • Wait for replication (w >=2)
- 22. Tagging • Control wheredata is written to, and read from • Each member can have one or more tags – tags: {dc: "ny"} – tags: {dc: "ny", subnet: "192.168", rack: "row3rk7"} • Replica set defines rules for write concerns • Rules can change without changing app code
- 23. { _id : "mySet", members: [ {_id : 0, host : "A", tags : {"dc": "ny"}}, {_id : 1, host : "B", tags : {"dc": "ny"}}, {_id : 2, host : "C", tags : {"dc": "sf"}}, {_id : 3, host : "D", tags : {"dc": "sf"}}, {_id : 4, host : "E", tags : {"dc": "cloud"}}], settings : { getLastErrorModes : { allDCs : {"dc" : 3}, someDCs : {"dc" : 2}} } } > db.blogs.insert({...}) > db.runCommand({getLastError : 1, w : "someDCs"}) Tagging Example
- 24. Read Preference Modes •5 modes – primary (only) - Default – primaryPreferred – secondary – secondaryPreferred – Nearest When more than one node is possible, closest node is used for reads (all modes but primary)
- 25. Tagged Read Preference •Custom read preferences • Control where you read from by (node) tags – E.g. { "disk": "ssd", "use": "reporting" } • Use in conjunction with standard read preferences – Except primary
- 26.
- 27.
- 28. Working Set ExceedsPhysical Memory
- 29.
- 30.
- 31. Partitioning • User definesshard key • Shard key defines range of data • Key space is like points on a line • Range is a segment of that line (chunk), smaller than 64MB • Chunks are migrated from one shard to another to maintain a balanced state
- 32. Shard Key • Shardkey is immutable • Shard key values are immutable • Shard key must be indexed • Shard key limited to 512 bytes in size • Shard key used to route queries – Choose a field commonly used in queries • Only shard key can be unique across shards
- 33. Shard Key Considerations •Cardinality • Write Distribution • Query Isolation • Reliability • Index Locality
- 34. Initially 1 chunk Defaultmax chunk size: 64mb MongoDB automatically splits & migrates chunks when max reached Data Distribution
- 35. Queries routed tospecific shards MongoDB balances cluster MongoDB migrates data to new nodes Routing and Balancing
- 36. Partitioning - ∞ +∞ shard 2 shard 3
- 37. Partitioning - ∞ +∞ - ∞ { x : 1}, { x : 3} …. { x : 99} + ∞ shard 2 shard 3 shard 2 shard 3
- 38. Partitioning - ∞ +∞ - ∞ { x : 1}, { x : 3} …. { x : 99} + ∞ - ∞ { x : 1} …. { x : 55} { x : 56} …. { x : 110} + ∞ shard 2 shard 3 shard 2 shard 3 shard 2 shard 3
- 39. Partitioning - ∞ +∞ - ∞ { x : 1}, { x : 3} …. { x : 99} + ∞ - ∞ { x : 1} …. { x : 55} { x : 56} …. { x : 110} + ∞ shard 2 shard 3 shard 2 shard 3 shard 2 shard 3
- 40. Partitioning - ∞ +∞ - ∞ { x : 1}, { x : 3} …. { x : 99} + ∞ - ∞ { x : 1} …. { x : 55} { x : 56} …. { x : 110} + ∞ shard 2 shard 3 shard 2 shard 3 shard 2 shard 3
- 41. MongoDB Auto-Sharding • Minimaleffort required – Same interface as single mongod • Two steps – Enable Sharding for a database – Shard collection within database
- 42.
- 43. What is aShard? • Shard is a node of the cluster • Shard can be a single mongod or a replica set
- 44. Meta Data Storage •Config Server – Stores cluster chunk ranges and locations – Can have only 1 or 3 (production must have 3) – Not a replica set
- 45. Routing and ManagingData • Mongos – Acts as a router / balancer – No local data (persists to config database) – Can have 1 or many
- 46.
- 47.
- 48.
- 49. mongod --configsvr Starts aconfiguration server on the default port (27019) Starting the Configuration Server
- 50. mongos --configdb <hostname>:27019 For3 configuration servers: mongos --configdb<host1>:<port1>,<host2>:<port2>,<host3>:<port3> Thisis always how to start a new mongos, even if the cluster is already running Start the mongos Router
- 51. mongod --shardsvr Starts amongod with the default shard port (27018) Shard is not yet connected to the rest of the cluster Shard may have already been running in production Start the shard database
- 52. On mongos: – sh.addShard(„<host>:27018‟) Addinga replica set: – sh.addShard(„<rsname>/<seedlist>‟) Add the Shard
- 53. db.runCommand({ listshards:1 }) {"shards" : [{"_id”:"shard0000”,"host”:”<hostname>:27018”} ], "ok" : 1 } Verify that the shard was added
- 54. Enabling Sharding • Enablesharding on a database sh.enableSharding(“<dbname>”) • Shard a collection with the given key sh.shardCollection(“<dbname>.people”,{“country”:1}) • Use a compound shard key to prevent duplicates sh.shardCollection(“<dbname>.cars”,{“year”:1,”uniqueid”:1})
- 55. Tag Aware Sharding •Tag aware sharding allows you to control the distribution of your data • Tag a range of shard keys – sh.addTagRange(<collection>,<min>,<max>,<tag>) • Tag a shard – sh.addShardTag(<shard>,<tag>)
- 56.
- 57. Cluster Request Routing •Targeted Queries • Scatter Gather Queries • Scatter Gather Queries with Sort
- 58. Cluster Request Routing:Targeted Query
- 59.
- 60. Request routed toappropriate shard
- 61.
- 62. Mongos returns resultsto client
- 63. Cluster Request Routing:Non-Targeted Query
- 64.
- 65. Request sent toall shards
- 66. Shards return resultsto mongos
- 67. Mongos returns resultsto client
- 68. Cluster Request Routing:Non-Targeted Query with Sort
- 69. Non-Targeted request withsort received
- 70. Request sent toall shards
- 71. Query and sortperformed locally
- 72. Shards return resultsto mongos
- 73. Mongos merges sortedresults
- 74. Mongos returns resultsto client
- 75.
- 76. 76 • Replica setpour haute disponibilité • Sharding pour montée en charge • Write concern • Clé de sharding Résumé
- 77. 77 – Backup – Reprisesur incident Prochaine session – 3 Juin
Editor's Notes
- #10 Basic explanation2 or more nodes form the setQuorum
- #11 Initialize -> ElectionPrimary + data replication from primary to secondaryHeartbeat every 2 seconds, timeout 10 seconds
- #12 Primary down/network failureAutomatic election of new primary if majority existsFailover usually takes a couple of seconds. Depending on your application code and configuration, this can be seamless/transparent.
- #13 New primary electedReplication established from new primary
- #14 Down node comes upRejoins setsRecovery and then secondary
- #15 Note that replication doesn’t always need to pull from the primary. Will pull from secondary if it is faster (less ping time).
- #17 PrimaryData memberSecondaryHot standbyArbitersVoting member
- #18 PriorityFloating point number between 0..1000Highest member that is up to date wins Up to date == within 10 seconds of primaryIf a higher priority member catches up, it will force election and win Slave DelayLags behind master by configurable time delay Automatically hidden from clientsProtects against operator errorsFat fingeringApplication corrupts data
- #19 ConsistencyWrite preferencesRead preferences
- #24 Using 'someDCs' so that in the event of an outage, at least a majority of the DCs would receive the change. This favors availability over durability.
- #29 Indexes should be contained in working set.
- #30 From mainframes, to RAC Oracle servers... People solved problems by adding more resources to a single machine.
- #31 Large scale operation can be combined with high performance on commodity hardware through horizontal scalingBuild - Document oriented database maps perfectly to object oriented languagesScale - MongoDB presents clear path to scalability that isn't ops intensive - Provides same interface for sharded cluster as single instance
- #33 _id could be unique across shards if used as shard key.we could only guarantee uniqueness of (any) attributes if the keys are used as shard keys with unique attribute equals true
- #34 Cardinality – Can your data be broken down enough?Query Isolation - query targeting to a specific shardReliability – shard outagesA good shard key can:Optimize routingMinimize (unnecessary) trafficAllow best scaling
- #49 Don’t use this setup in production!Only one Config server (No Fault Tolerance)Shard not in a replica set (Low Availability)Only one mongos and shard (No Performance Improvement)Useful for development or demonstrating configuration mechanics
- #53 MongoDB 2.2 and later only need <host> and <port> for one member of the replica set
- #56 This can be skipped for the intro talk, but might be good to include if you’re doing the combined sharding talk. Totally optional, you don’t really have enough time to do this topic justice but it might be worth a mention.
- #74 The mongos does not have to load the whole set into memory since each shard sorts locally. The mongos can just getMore from the shards as needed and incrementally return the results to the client.