MongoDB World 2016 : Advanced Aggregation
Download as PPTX, PDF0 likes401 views
This document discusses MongoDB's aggregation framework and provides an example of creating a summary of test results from a public MOT (Ministry of Transport) dataset containing over 25 million records. It shows how to use aggregation pipeline stages like $match, $project, $group to filter the data to only cars from 2013, calculate the age of each car, and then group the results to output statistics on counts, average mileages, and number of passes for each make and age combination. The aggregation framework allows processing large collections in parallel and creating new data from existing data.
![#MDBW16
Use $match to filter
cars = { "$match" : { "TestClassID" : { "$eq" : "4" }}}
motorcycles = {
"$match" : {
"$or" : [
{
"TestClassID" : "1"
},
{
"TestClassID" : "2"
}
]
}
}](https://image.slidesharecdn.com/mdbw-2016-aggregation-160719171350/85/MongoDB-World-2016-Advanced-Aggregation-10-320.jpg)
![#MDBW16
Lets make a collection of Cars
removeNulls = { "$match" : { "FirstUseDate" : { "$ne" : "NULL" }}}
carsonly = { "$match" : { "TestClassID" : â4â}}
output = { â$outâ : âcars_2013â }
db.results_2013.aggregate( [ removeNulls, carsonly, output ] )](https://image.slidesharecdn.com/mdbw-2016-aggregation-160719171350/85/MongoDB-World-2016-Advanced-Aggregation-11-320.jpg)
![#MDBW16
We $project to get the data we want
ageinusecs ={ "$subtract" :[ "$TestDate", "$FirstUseDateâ]}
ageinyears = { "$divide" :[ ageinusecs,(1000*3600*24*365)]}
floorage = { "$floor" : ageinyears }
ispass = { "$cond" : [{"$eq": ["$TestResult","P"]},1,0]}](https://image.slidesharecdn.com/mdbw-2016-aggregation-160719171350/85/MongoDB-World-2016-Advanced-Aggregation-13-320.jpg)
![#MDBW16
Now put it all in pipeline
db.results_2013.aggregate([project,group,out])
out = { "$out" : âcars_summary" }
The $out operator creates a new collection](https://image.slidesharecdn.com/mdbw-2016-aggregation-160719171350/85/MongoDB-World-2016-Advanced-Aggregation-16-320.jpg)
MongoDB World 2016 : Advanced Aggregation
- 1. Advanced MongoDB Aggregation Joe Drumgoole Director of Developer Advocacy, EMEA @jdrumgoole
- 2. #MDBW16 MongoDB Aggregation Framework âą One of the least understood parts of MongoDB âą Many customers never get to the aggregation framework âą Had limited utility 2.6 (16MB limit) âą The aggregation framework âgrew upâ in 3.0 âą We continue to enhance (3.2 and now 3..4)
- 3. #MDBW16 A quick reminder âą A Processing Pipeline âą Design to process large groups of documents in parallel âą Is shard aware âą Can create new data from old
- 4. #MDBW16 A Typical Pipeline Match Project Group Sort Find Query Select Fields Combine Fields Rename fields Calculate Group By Execute accumulators Rename fields Sort results
- 5. #MDBW16 Description of Data Set âą MOT : Ministry of Transport, then âą VOSA : Vehicle and Operator Services Agency and then âą DVSA : Driver and Vehicle Services Agency âą The test is still called the MOT Test âą It is a test of road worthiness âą Nearly every motorised vehicle must pass an MOT every year âą New cars are exempt for three years
- 6. #MDBW16 Introduction to the MOT Public Data Set
- 7. #MDBW16 This is a big Collection > db.test_results.count() 253472477 > 253 million records
- 8. #MDBW16 Example Document { "_id" : ObjectId("5759ee6e8684975e1098af68"), "TestID" : 400, "VehicleID" : "278", "TestDate" : ISODate("2013-04-23T00:00:00Z"), "TestClassID" : "4", "TestType" : "N", "TestResult" : "P", "TestMileage" : 99284, "Postcode" : "E", "Make" : "AUDI", "Model" : "A3", "Colour" : "BLACK", "FuelType" : "P", "CylinderCapacity" : 1598, "FirstUseDate" : ISODate("2003-11-11T00:00:00Zâ) }
- 9. #MDBW16 We will work with one year : 2013 > db.results_2013.count() 37390457 Still 37.3 million records
- 10. #MDBW16 Use $match to filter cars = { "$match" : { "TestClassID" : { "$eq" : "4" }}} motorcycles = { "$match" : { "$or" : [ { "TestClassID" : "1" }, { "TestClassID" : "2" } ] } }
- 11. #MDBW16 Lets make a collection of Cars removeNulls = { "$match" : { "FirstUseDate" : { "$ne" : "NULL" }}} carsonly = { "$match" : { "TestClassID" : â4â}} output = { â$outâ : âcars_2013â } db.results_2013.aggregate( [ removeNulls, carsonly, output ] )
- 12. #MDBW16 Now lets create a Summary For each car of a given make and age: âą The total number of cars âą The average mileage for this collection of cars âą The total number of passes
- 13. #MDBW16 We $project to get the data we want ageinusecs ={ "$subtract" :[ "$TestDate", "$FirstUseDateâ]} ageinyears = { "$divide" :[ ageinusecs,(1000*3600*24*365)]} floorage = { "$floor" : ageinyears } ispass = { "$cond" : [{"$eq": ["$TestResult","P"]},1,0]}
- 14. #MDBW16 The actual $projection project = { "$project" : { "Makeâ :1, "VehicleID" :1, "TestResultâ :1, "TestDateâ :1, "TestMileageâ :1, "FirstUseDateâ :1, "Ageâ :floorage, "passâ :ispass }}
- 15. #MDBW16 We Then $group to get the accumulated values group = { "$group" : { "_id" : { "make": "$Make", "age" : "$Age" }, "count" : {"$sum":1} , "milesâ : {"$avg":"$TestMileage"}, "passes": {"$sum":"$passâ }}}
- 16. #MDBW16 Now put it all in pipeline db.results_2013.aggregate([project,group,out]) out = { "$out" : âcars_summary" } The $out operator creates a new collection
- 17. #MDBW16 Restrictions âą 16MB result limit if not using cursors or $out âą 100MB in memory limit unless $allowDiskUse is specified âą $geoNear must be the first operator in a pipeline âą $out must be the last operator in the pipeline
- 18. #MDBW16 Market Size $36 Billion Partners 1,000+ International Offices 15 Global Employees 575+ Downloads Worldwide 15,000,000+ Make a GIANT Impact www.mongodb.com/careers