MongoDB Aggregation Framework in action !
5 likes7,269 views
The document discusses MongoDB's aggregation framework. It provides examples of using aggregation operations like $group, $match, $sort, $limit, $project, and $unwind to summarize and analyze data. Specifically, it shows examples of counting campgrounds by rating, finding the top cities for campgrounds, finding the most appearing comic book characters, and analyzing data on "geeks" in Paris. The document also discusses how indexes can optimize aggregation pipelines and how pipelines can be split between shards for performance.
![Aggregation Pipeline
db.collection.aggregate(
[
stage1,
stage2,
...,
stageN
]
)](https://image.slidesharecdn.com/nantesmug-mongodbaggregationframework-150121023633-conversion-gate02/85/MongoDB-Aggregation-Framework-in-action-8-320.jpg)
![Campings
db.campings.aggregate([
{$group : {_id : "$ranking", total : {$sum : 1}}}
])
{ "_id" : "5 étoiles", "total" : 177 }
{ "_id" : "4 étoiles", "total" : 940 }
{ "_id" : "3 étoiles", "total" : 2224 }
{ "_id" : "2 étoiles", "total" : 1681 }
{ "_id" : "1 étoile", "total" : 389 }](https://image.slidesharecdn.com/nantesmug-mongodbaggregationframework-150121023633-conversion-gate02/85/MongoDB-Aggregation-Framework-in-action-14-320.jpg)
![Campings
db.campings.aggregate([
{$group : {_id : "$city", total : {$sum : 1}}},
{$sort : {total : -1}},
{$limit : 5},
{$project: {_id: 0, city : "$_id", total: 1}}
])
{ "city" : "ARGELÈS-SUR-MER" , "total" : 29 }
{ "city" : "AGDE", "total" : 23 }
{ "city" : "VIAS", "total" : 20 }
{ "city" : "SAINT-JEAN-DE-MONTS" , "total" : 20 }
{ "city" : "LES MATHES", "total" : 17 }](https://image.slidesharecdn.com/nantesmug-mongodbaggregationframework-150121023633-conversion-gate02/85/MongoDB-Aggregation-Framework-in-action-16-320.jpg)
![Campings
db.campings.aggregate([
{$group : {_id : "$city", total : {$sum : 1}}},
{$match : {total : 1}},
{$group: {_id: null, count: {$sum: 1 }}},
{$project: {_id: 0, count: 1}}
])
{ "count" : 2802 }](https://image.slidesharecdn.com/nantesmug-mongodbaggregationframework-150121023633-conversion-gate02/85/MongoDB-Aggregation-Framework-in-action-18-320.jpg)
![Marvel
{
"_id" : 4,
"title" : "Rogue (2004) #5",
"description" : "...",
"format" : "Comic",
"creators" : {
"available" : 6,
"items" : [...]
},
"characters" : {
"available" : 1,
"items" : [
{
"name" : "Rogue"
}
]
},
...
}
developer.marvel.com](https://image.slidesharecdn.com/nantesmug-mongodbaggregationframework-150121023633-conversion-gate02/85/MongoDB-Aggregation-Framework-in-action-19-320.jpg)
![Marvel
“Deconstructs an array field from the input documents to output a
document for each element.”
$unwind
{
"name" : "john doe",
"tags" : ["A", "B", "C"]
}
{
"name" : "john doe",
"tags" : "A"
}
{
"name" : "john doe",
"tags" : "B"
}
{
"name" : "john doe",
"tags" : "C"
}
$unwind](https://image.slidesharecdn.com/nantesmug-mongodbaggregationframework-150121023633-conversion-gate02/85/MongoDB-Aggregation-Framework-in-action-21-320.jpg)
![Marvel
db.comics.aggregate([
{$match : {"characters.returned" : {$gt : 0}}},
{$project : {title : 1, characters : 1}},
{$unwind : "$characters.items"},
{$group : {_id : "$characters.items.name", total : {$sum : 1}}},
{$sort : {total : -1}},
{$limit : 5}
])
{"_id": "Spider-Man","total": 2413}
{"_id": "X-Men","total": 2320}
{"_id": "Iron Man","total": 1904}
{"_id": "Wolverine","total": 1594}
{"_id": "Captain America" ,"total": 1367}](https://image.slidesharecdn.com/nantesmug-mongodbaggregationframework-150121023633-conversion-gate02/85/MongoDB-Aggregation-Framework-in-action-22-320.jpg)
![Marvel
db.comics.aggregate([
{$match : {"characters.returned" : {$gt : 0}}},
{$project : {title : 1, characters : 1}},
{$unwind : "$characters.items"},
{$group : {
_id : "$characters.items.name",
total : {$sum : 1},
comics : {$push : {id : "$_id", title : "$title"}}}
},
{$out : "characters"}
])](https://image.slidesharecdn.com/nantesmug-mongodbaggregationframework-150121023633-conversion-gate02/85/MongoDB-Aggregation-Framework-in-action-24-320.jpg)
![Marvel
{
"_id": "Frog-Man",
"total": 2,
"comics": [
{
"id": 38126,
"title": "Spider-Man: New York Stories (Trade Paperback)"
},
{
"id": 39753,
"title": "Spider-Island: Avengers (2011) #1"
}
]
}](https://image.slidesharecdn.com/nantesmug-mongodbaggregationframework-150121023633-conversion-gate02/85/MongoDB-Aggregation-Framework-in-action-25-320.jpg)
![Geeks
{
"_id" : ObjectId("54bd3f1a84c2c169160a88d5"),
"nom" : "Mark Zuckerberg",
"ville" : "Palo Alto",
"likes" : [
"Facebook",
"Tongues",
"PHP"
],
"imageUrl" : "static/GIT_HASH/img/geek5.jpg",
"location" : {
"type" : "Point",
"coordinates" : [
-122.1430195,
37.4418834
]
}
}
code-story.net](https://image.slidesharecdn.com/nantesmug-mongodbaggregationframework-150121023633-conversion-gate02/85/MongoDB-Aggregation-Framework-in-action-26-320.jpg)
![Geeks
db.geeks.aggregate([
{
$geoNear: {
near: {
type: "Point",
coordinates: [2.352241, 48.856638]
},
distanceField: "distance",
spherical: true,
maxDistance: 10000
}
},
...
])](https://image.slidesharecdn.com/nantesmug-mongodbaggregationframework-150121023633-conversion-gate02/85/MongoDB-Aggregation-Framework-in-action-28-320.jpg)
![Geeks
{
"_id" : "java",
"total" : 8,
"friends" : [
{
"nom" : "...",
"distance" : 3.015741215085397
},
{
"nom" : "...",
"distance" : 9864.484291991206
},
...
]
}](https://image.slidesharecdn.com/nantesmug-mongodbaggregationframework-150121023633-conversion-gate02/85/MongoDB-Aggregation-Framework-in-action-29-320.jpg)
![Index
● Index utilisé pour certains opérateurs, en
début de pipeline
○ $match
○ $sort
○ $geoNear
● explain
○ db.collection.aggregate([...],
{explain: true})](https://image.slidesharecdn.com/nantesmug-mongodbaggregationframework-150121023633-conversion-gate02/85/MongoDB-Aggregation-Framework-in-action-30-320.jpg)
MongoDB Aggregation Framework in action !
- 1. MongoDB Aggregation Framework in action ! MongoDB User Group - Nantes mardi 20 janvier 2015 @sebprunier
- 2. /me Sébastien Prunier Développeur chez SERLI @sebprunier http://blog.sebprunier.com
- 5. MongoDB Aggregation Framework Opérations simples Map-Reduce Aggregation Pipeline
- 10. Opérateurs vs. SQL SQL Terms, Functions, and Concepts MongoDB Aggregation Operators WHERE $match GROUP BY $group HAVING $match SELECT $project ORDER BY $sort LIMIT $limit SUM() $sum COUNT() $sum join N/A ~ $unwind / arrays
- 12. Campings { "_id" : ObjectId("54bd347880d46d750f7a48c2"), "raking_date" : ISODate("2013-06-06T22:00:00Z"), "publication_date" : ISODate("2013-06-06T22:00:00Z"), "typology" : "CAMPING", "ranking" : "2 étoiles", "category" : "Tourisme", "mention" : "-", "commercial_name" : "CAMPING LA BERGERIE", "address" : "4231 route de Giens", "zip_code" : "83400", "city" : "HYÈRES", "phone" : "494589175", "email" : "[email protected]", "website" : "www.camping-de-la-bergerie.com", "visit_type" : "-", "capacity" : "-", "number_of_rooms" : "-", "number_of_spaces" : "55", "number_of_housing_units" : "-", "number_of_chambers" : "-" } data.gouv.fr
- 13. Campings “ Déterminer le nombre de campings pour chaque niveau de classement (1 étoile, 2 étoiles, etc…) ” $group
- 14. Campings db.campings.aggregate([ {$group : {_id : "$ranking", total : {$sum : 1}}} ]) { "_id" : "5 étoiles", "total" : 177 } { "_id" : "4 étoiles", "total" : 940 } { "_id" : "3 étoiles", "total" : 2224 } { "_id" : "2 étoiles", "total" : 1681 } { "_id" : "1 étoile", "total" : 389 }
- 15. Campings “ Top 5 des villes avec le plus de campings ” $group $sort $limit $project
- 16. Campings db.campings.aggregate([ {$group : {_id : "$city", total : {$sum : 1}}}, {$sort : {total : -1}}, {$limit : 5}, {$project: {_id: 0, city : "$_id", total: 1}} ]) { "city" : "ARGELÈS-SUR-MER" , "total" : 29 } { "city" : "AGDE", "total" : 23 } { "city" : "VIAS", "total" : 20 } { "city" : "SAINT-JEAN-DE-MONTS" , "total" : 20 } { "city" : "LES MATHES", "total" : 17 }
- 17. Campings “ Nombre de villes avec seulement un camping ” $group $match $project
- 18. Campings db.campings.aggregate([ {$group : {_id : "$city", total : {$sum : 1}}}, {$match : {total : 1}}, {$group: {_id: null, count: {$sum: 1 }}}, {$project: {_id: 0, count: 1}} ]) { "count" : 2802 }
- 19. Marvel { "_id" : 4, "title" : "Rogue (2004) #5", "description" : "...", "format" : "Comic", "creators" : { "available" : 6, "items" : [...] }, "characters" : { "available" : 1, "items" : [ { "name" : "Rogue" } ] }, ... } developer.marvel.com
- 20. Marvel “ Top 5 des personnages apparaissant dans le plus de bandes dessinées ” $match $project $unwind $group $sort $limit
- 21. Marvel “Deconstructs an array field from the input documents to output a document for each element.” $unwind { "name" : "john doe", "tags" : ["A", "B", "C"] } { "name" : "john doe", "tags" : "A" } { "name" : "john doe", "tags" : "B" } { "name" : "john doe", "tags" : "C" } $unwind
- 22. Marvel db.comics.aggregate([ {$match : {"characters.returned" : {$gt : 0}}}, {$project : {title : 1, characters : 1}}, {$unwind : "$characters.items"}, {$group : {_id : "$characters.items.name", total : {$sum : 1}}}, {$sort : {total : -1}}, {$limit : 5} ]) {"_id": "Spider-Man","total": 2413} {"_id": "X-Men","total": 2320} {"_id": "Iron Man","total": 1904} {"_id": "Wolverine","total": 1594} {"_id": "Captain America" ,"total": 1367}
- 23. Marvel “ Créez la collection des personnages à partir de la collections des bandes dessinées ” $match $project $unwind $group $out
- 24. Marvel db.comics.aggregate([ {$match : {"characters.returned" : {$gt : 0}}}, {$project : {title : 1, characters : 1}}, {$unwind : "$characters.items"}, {$group : { _id : "$characters.items.name", total : {$sum : 1}, comics : {$push : {id : "$_id", title : "$title"}}} }, {$out : "characters"} ])
- 25. Marvel { "_id": "Frog-Man", "total": 2, "comics": [ { "id": 38126, "title": "Spider-Man: New York Stories (Trade Paperback)" }, { "id": 39753, "title": "Spider-Island: Avengers (2011) #1" } ] }
- 26. Geeks { "_id" : ObjectId("54bd3f1a84c2c169160a88d5"), "nom" : "Mark Zuckerberg", "ville" : "Palo Alto", "likes" : [ "Facebook", "Tongues", "PHP" ], "imageUrl" : "static/GIT_HASH/img/geek5.jpg", "location" : { "type" : "Point", "coordinates" : [ -122.1430195, 37.4418834 ] } } code-story.net
- 27. Geeks “ Rechercher ce qui rassemble le plus de geeks autour de Paris ” $geoNear $project $unwind $group $match $sort
- 28. Geeks db.geeks.aggregate([ { $geoNear: { near: { type: "Point", coordinates: [2.352241, 48.856638] }, distanceField: "distance", spherical: true, maxDistance: 10000 } }, ... ])
- 29. Geeks { "_id" : "java", "total" : 8, "friends" : [ { "nom" : "...", "distance" : 3.015741215085397 }, { "nom" : "...", "distance" : 9864.484291991206 }, ... ] }
- 30. Index ● Index utilisé pour certains opérateurs, en début de pipeline ○ $match ○ $sort ○ $geoNear ● explain ○ db.collection.aggregate([...], {explain: true})
- 31. Index ● Optimisations { $sort: { age : -1 } }, { $match: { status: 'A' } } { $match: { status: 'A' } }, { $sort: { age : -1 } } “ Optimizations are subject to change between releases. “
- 32. Sharding ● Découpage du pipeline en deux parties ○ 1ère partie (jusqu’au premier $group ou $sort) exécutée sur chaque shard (certains shards peuvent être exclus grâce à un $match) ○ 2ème partie exécutée sur le shard primaire à partir des résultats consolidés de la première partie
- 33. The end Merci pour votre attention ! ● Questions / Réponses ● Partage d’expérience