SlideShare a Scribd company logo
asya999
AGGREGATION
How MongoDB 4.2 Pipeline is Powering Queries, Updates and Views
PIPELINE POWER+++
asya999
AGGREGATION
How MongoDB 4.2 Pipeline is Powering Queries, Updates and Views
PIPELINE POWER+++
PREVIOUSLY ...
... 2017
#MDBW17
Analytics with MongoDB Aggregation Framework
@asya999 by Asya Kamsky,
Lead MongoDB Maven
PIPELINE POWER
STORE
RETRIEVE
ps ax | grep mongod | head 1
*nix command line pipe
PIPELINE
$match $group | $sort|
Input stream {} {} {} {} Result {} {} ...
PIPELINE
MongoDB document pipeline
DATA PIPELINE
STAGES
Stage 1 Stage 2 Stage 3 Stage 4
{} {} {} {}
{} {} {} {}
DATA PIPELINE
{} {} {} {}
{"$stage":{ ... }}
START
Collection
View
Special stage
STAGES
{title: "The Great Gatsby",
language: "English",
subjects: "Long Island"}
{title: "The Great Gatsby",
language: "English",
subjects: "New York"}
{title: "The Great Gatsby",
language: "English",
subjects: "1920s"}
{title: "The Great Gatsby",
language: "English",
subjects: [
"Long Island",
"New York",
"1920s"] },
{"$match":{"language":"English"}}
$match
{ _id:"Long Island",
count: 1 },
$group
{ _id: "New York",
count: 2 },
$unwind
{ _id: "1920s",
count: 1 },
$sort $skip$limit $project
{"$unwind":"$subjects"}
{"$group":{"_id":"$subjects", "count":{"$sum:1}}
{ _id: "Harlem",
count: 1 },
{ _id: "Long Island",
count: 1 },
{ _id: "New York",
count: 2 },
{ _id: "1920s",
count: 1 },
{title: "Open City",
language: "English",
subjects: [
"New York"
"Harlem" ] }
{ title: "The Great Gatsby",
language: "English",
subjects: [
"Long Island",
"New York",
"1920s"] },
{ title: "War and Peace",
language: "Russian",
subjects: [
"Russia",
"War of 1812",
"Napoleon"] },
{ title: "Open City",
language: "English",
subjects: [
"New York",
"Harlem" ] },
{title: "Open City",
language: "English",
subjects: "New York"}
{title: "Open City",
language: "English",
subjects: "Harlem"}
{ _id: "Harlem",
count: 1 },
{"$sort:{"count":-1} {"$limit":3}
{"$project":...}
MongoDB World 2019: Aggregation Pipeline Power++: How MongoDB 4.2 Pipeline Empowers Queries, Updates, and Materialized Views [MongoDB]
$cursor
{title: "The Great Gatsby",
language: "English".
subjects: "Long Island"}
{title: "The Great Gatsby",
language: "English",
subjects: "New York"}
{title: "The Great Gatsby",
language: "English",
subjects: "1920s"}
{title: "The Great Gatsby",
language: "English",
subjects: [
"Long Island",
"New York",
"1920s"] },
{"$match":{"language":"English"}}
$match
{ _id:"Long Island",
count: 1 },
$group
{ _id: "New York",
count: 2 },
$unwind
{ _id: "1920s",
count: 1 },
$sort $skip$limit $project
{"$unwind":"$subjects"}
{"$group":{"_id":"$subjects", "count":{"$sum:1}}
{ _id: "Harlem",
count: 1 },
{ _id: "Long Island",
count: 1 },
{ _id: "New York",
count: 2 },
{ _id: "1920s",
count: 1 },
{title: "Open City",
language: "English",
subjects: [
"New York"
"Harlem" ] }
{ title: "The Great Gatsby",
language: "English",
subjects: [
"Long Island",
"New York",
"1920s"] },
{ title: "War and Peace",
language: "Russian",
subjects: [
"Russia",
"War of 1812",
"Napoleon"] },
{ title: "Open City",
language: "English",
subjects: [
"New York",
"Harlem" ] },
{title: "Open City",
language: "English",
subjects: "New York"}
{title: "Open City",
language: "English",
subjects: "Harlem"}
{ _id: "Harlem",
count: 1 },
{"$sort:{"count":-1} {"$limit":3}
{"$project":...}
$group $sort
1
INPUT STAGE RESULTSSTAGE
STREAMING RESOURCE USE
Each document is streamed through in RAM
INPUT STAGE RESULTSSTAGE
BLOCKING RESOURCE USE
Everything has to be kept in RAM (or spill)
MongoDB World 2019: Aggregation Pipeline Power++: How MongoDB 4.2 Pipeline Empowers Queries, Updates, and Materialized Views [MongoDB]
MongoDB World 2019: Aggregation Pipeline Power++: How MongoDB 4.2 Pipeline Empowers Queries, Updates, and Materialized Views [MongoDB]
MongoDB World 2019: Aggregation Pipeline Power++: How MongoDB 4.2 Pipeline Empowers Queries, Updates, and Materialized Views [MongoDB]
MongoDB World 2019: Aggregation Pipeline Power++: How MongoDB 4.2 Pipeline Empowers Queries, Updates, and Materialized Views [MongoDB]
MongoDB World 2019: Aggregation Pipeline Power++: How MongoDB 4.2 Pipeline Empowers Queries, Updates, and Materialized Views [MongoDB]
$sort$match $group
start=ISODate("...")
end=ISODate("...")
{
user: "303900",
ipaddr: "71.56.112.56",
ts:ISODate("2017-05-08T...")
}
{$match:{ts:{$gte:start,$lt:end}}},
{$sort:{ts:1}},
{$group:{_id:"$user",ips:{$push:{ip:"$ipaddr", ts:"$ts"}},
diffIps:{$addToSet:"$ipaddr"}}},
{$match:{"diffIps.1":{$exists:true}}},
{$addFields:{diffs: {$filter:{
input:{$map:{
input: {$range:[0,{$subtract:[{$size:"$ips"},1]}]}, as:"i",
in:{$let:{vars:{ip1:{$arrayElemAt:["$ips","$$i"]},
ip2:{$arrayElemAt:["$ips",{$add:["$$i",1]}]}},
in:{
diff:{$cond:{
if:{$ne:["$$ip1.ip","$$ip2.ip"]},
then:{$divide:[{$subtract:["$$ip2.ts","$$ip1.ts"]},60000]},
else: 9999 }},
ip1:"$$ip1.ip", t1:"$$ip1.ts",
ip2:"$$ip2.ip", t2:"$$ip2.ts"
}}}}},
cond:{$lt:["$$this.diff",10]}
}}}},
{$match:{"diffs":{$ne:[]}}},
{$project:{_id:0, user:"$_id", suspectLogins:"$diffs"}}
{ "user" : "35237073",
"suspectLogins" : [
{"diff": 4.8333333333,
"ip1": "106.220.151.16",
"t1":"2017-05-08T06:58",
"ip2": "223.182.113.15"
"t2":"2017-05-08T07:03"
},
{"diff": 8.3,
"ip1": "223.182.113.15",
"t1":"2017-05-08T07:03",
"ip2": "49.206.217.26",
"t2":"2017-05-08T07:11"
}
]
} $match $addFields $match $project
https://github.com/asya999/mdbw17
5 minute review
https://github.com/asya999/mdbw17
#MDBW17
Analytics with MongoDB Aggregation Framework
@asya999 by Asya Kamsky,
Lead MongoDB Maven
PIPELINE POWER
https://github.com/asya999/mdbw17
PREVIOUSLY ...
... 2017
PREVIOUSLY ...
... 2017 ...
PREVIOUSLY ...
... 2017 ... 2018
THE FUTURE OF AGGREGATION
Better performance & optimizations
More stages & expressions
More options for output
Compass helper for aggregate
Unify different languages
THE FUTURE OF AGGREGATION
Better performance & optimizations
More stages & expressions
More options for output
Compass helper for aggregate
Unify different languages
THE FUTURE OF AGGREGATION
Better performance & optimizations
More stages & expressions
More options for output
Compass helper for aggregate
Unify different languages
THE FUTURE OF AGGREGATION
More options for output
Unify different languages
More options for output
Unify different languages
THE PRESENT OF AGGREGATION
Unify Different Languages
{children: [
{name:"Max", dob:"1994-12-01", dep:true},
{name:"Sam", dob:"1997-09-28", dep:true},
{name:"Kim", dob:"2000-02-29", dep:true}
]}
AGGREGATION
Unify Different Languages
{children: [
{name:"Max", dob:"1994-12-01", dep:true},
{name:"Sam", dob:"1997-09-28", dep:true},
{name:"Kim", dob:"2000-02-29", dep:true}
]}
AGGREGATION
Unify Different Languages
db.c.aggregate([
{$addFields:{
numChildren:{$size:"$children"},
numDependents:{$size:{
$filter:{
input:"$children.dep",
cond: "$$this"
}
}}
}},
...
])
{children: [
{name:"Max", dob:"1994-12-01", dep:true},
{name:"Sam", dob:"1997-09-28", dep:true},
{name:"Kim", dob:"2000-02-29", dep:true}
]}
AGGREGATION
FIND
Unify Different Languages
db.c.aggregate([
{$addFields:{
numChildren:{$size:"$children"},
numDependents:{$size:{
$filter:{
input:"$children.dep",
cond: "$$this"
}
}}
}},
...
])
{children: [
{name:"Max", dob:"1994-12-01", dep:true},
{name:"Sam", dob:"1997-09-28", dep:true},
{name:"Kim", dob:"2000-02-29", dep:true}
]}
AGGREGATION
FIND
Unify Different Languages
db.c.find (
{$expr:{
$lt:[
{$size:{$filter:{
input: "$children.dep",
cond: "$$this"
}}},
2
]
}}
)
{children: [
{name:"Max", dob:"1994-12-01", dep:true},
{name:"Sam", dob:"1997-09-28", dep:true},
{name:"Kim", dob:"2000-02-29", dep:true}
]}
AGGREGATION
FIND
UPDATE
Unify Different Languages
db.c.find (
{$expr:{
$lt:[
{$size:{$filter:{
input: "$children.dep",
cond: "$$this"
}}},
2
]
}}
)
{children: [
{name:"Max", dob:"1994-12-01", dep:true},
{name:"Sam", dob:"1997-09-28", dep:true},
{name:"Kim", dob:"2000-02-29", dep:true}
]}
AGGREGATION
FIND
UPDATE
Unify Different Languages
db.c.update(
{$expr:{
$anyElementTrue:{$map:{
input:"$children",
in: {$and:[
{$lt:["$$this.dob","1997-01-22"]},
"$$this.dep"
]}
}}
}},
{$set:{ audit:true }}
)
db.coll.update(
<query>,
<update>,
<options>
)
Update
db.coll.update(
<query>,
<update>,
<options>
)
Update
db.coll.update(
<query>,
<update>,
<options>
)
<update>
Update
<update>
{
$set: { },
$inc: { },
$...
}
Update
{
f1: <value>,
f2: <value>,
...
}
<update>
{ } OR [ ]
Update in 4.2
[ ]
<update>
[ <aggregation-pipeline> ]
Update in 4.2
[ ]
Updates Using
Aggregation Pipeline
{ $addFields: { } }
{ $project: { } }
{ $replaceRoot: { } }
{ $set: { } }
{ $unset: [ ] }
{ $replaceWith: { } }
{ _id: 1 }
{ _id: 1, a: 10 }
{ _id: 1, a: 100 }
---
{ _id: 1, a: "10" }
db.coll.update({_id:1},
{$inc:{a:1}},
{upsert:true})
{ _id: 1, a: 1 }
{ _id: 1, a: 11 }
{ _id: 1, a: 101 }
{ _id: 1, a: 1 }
"errmsg" : "Cannot apply
to a value of non-numeric type."
{ _id: 1 }
{ _id: 1, a: 10 }
{ _id: 1, a: 100 }
---
{ _id: 1, a: "10" }
db.coll.update({_id:1},
[ {$set:{a:{$sum:["$a",1]}}} ],
{upsert:true})
{ _id: 1, a: 1 }
{ _id: 1, a: 11 }
{ _id: 1, a: 101 }
{ _id: 1, a: 1 }
{ _id: 1, a: 1 }
{ _id: 1 }
{ _id: 1, a: 10 }
{ _id: 1, a: 100 }
---
{ _id: 1, a: "10" }
db.coll.update({_id:1},
[ {$set:{a:{$add:["$a",1]}}} ],
{upsert:true})
{ _id: 1, a: 1 }
{ _id: 1, a: 11 }
{ _id: 1, a: 101 }
{ _id: 1, a: 1 }
"errmsg" : "$add only supports
numeric or date types, not string"
{ _id: 1 }
{ _id: 1, a: 10 }
{ _id: 1, a: 100 }
---
{ _id: 1, a: "10" }
db.coll.update({_id:1},
[
],
{upsert:true})
{ _id:1, a: 21 }
{ _id: 1, a: 11 }
{ _id: 1, a: 101 }
{ _id:1, a: 21 }
{ _id: 1 }
{ _id: 1, a: 10 }
{ _id: 1, a: 100 }
---
{ _id: 1, a: "10" }
db.coll.update({_id:1},
[ {$set:{a:{$cond:{
if:
then: , else: }} }}]
{upsert:true})
{ _id:1, a: 21 }
{ _id: 1, a: 11 }
{ _id: 1, a: 101 }
{ _id:1, a: 21 }
{ _id: 1 }
{ _id: 1, a: 10 }
{ _id: 1, a: 100 }
---
{ _id: 1, a: "10" }
db.coll.update({_id:1},
[ {$set:{a:{$cond:{
if: {$eq:[{$type:"$a"},"missing"]},
then: , else: }} }}],
{upsert:true})
{ _id:1, a: 21 }
{ _id: 1, a: 11 }
{ _id: 1, a: 101 }
{ _id:1, a: 21 }
{ _id: 1 }
{ _id: 1, a: 10 }
{ _id: 1, a: 100 }
---
{ _id: 1, a: "10" }
db.coll.update({_id:1},
[ {$set:{a:{$cond:{
if: {$eq:[{$type:"$a"},"missing"]},
then: 21, else: {$sum:["$a", 1]} }} }}],
{upsert:true})
{ _id:1, a: 21 }
{ _id: 1, a: 11 }
{ _id: 1, a: 101 }
{ _id:1, a: 21 }
{ _id: 1 }
{ _id: 1, a: 10 }
{ _id: 1, a: 100 }
---
{ _id: 1, a: "10" }
db.coll.update({_id:1},
[ {$set:{a:{$cond:{
if: {$eq:[{$type:"$a"},"missing"]},
then: 21, else: {$sum:["$a", 1]} }} }}],
{upsert:true})
{ _id:1, a: 21 }
{ _id: 1, a: 11 }
{ _id: 1, a: 100 }
{ _id:1, a: 21 }
{ _id: 1 }
{ _id: 1, a: 10 }
{ _id: 1, a: 100 }
---
{ _id: 1, a: "10" }
db.coll.update({_id:1},
[ {$set:{a:{$min:[ 100, {$cond:{
if: {$eq:[{$type:"$a"},"missing"]},
then: 21, else: {$sum:["$a", 1]} }}]} }}],
{upsert:true})
{ _id:1, a: 21 }
{ _id: 1, a: 11 }
{ _id: 1, a: 100 }
{ _id:1, a: 21 }
{ _id: 1 }
{ _id: 1, a: 10 }
{ _id: 1, a: 100 }
---
{ _id: 1, a: "10" }
db.coll.update({_id:1},
[ {$set:{a:{$min:[ 100, {$cond:{
if: {$eq:[{$type:"$a"},"missing"]},
then: 21, else: {$sum:["$a", 1]} }}]} }}],
{upsert:true})
{ _id:1, a: 21 }
{ _id: 1, a: 11 }
{ _id: 1, a: 100 }
{ _id:1, a: 21 }
{ _id:1, a: 1 }
{ _id: 1 }
{ _id: 1, a: 10 }
{ _id: 1, a: 100 }
---
{ _id: 1, a: "10" }
db.coll.update({_id:1},
[ {$set:{a:{$min:[ 100, {$cond:{
if: {$eq:[{$type:"$a"},"missing"]},
then: 21, else: {$sum:["$a", 1]} }}]},
prev_a: "$a"}}], {upsert:true})
{ _id:1, a: 21 }
{ _id: 1, a: 11 }
{ _id: 1, a: 100 }
{ _id:1, a: 21 }
{ _id:1, a: 1 }
{ _id: 1 }
{ _id: 1, a: 10 }
{ _id: 1, a: 100 }
---
{ _id: 1, a: "10" }
db.coll.update({_id:1},
[ {$set:{a:{$min:[ 100, {$cond:{
if: {$eq:[{$type:"$a"},"missing"]},
then: 21, else: {$sum:["$a", 1]} }}]},
prev_a: "$a"}}], {upsert:true})
{ _id:1, a: 21 }
{ _id: 1, a: 11, prev_a: 10 }
{ _id: 1, a: 100, prev_a: 100 }
{ _id:1, a: 21 }
{ _id:1, a: 1, prev_a: "10" }
Set Defaults
{_id: 1, a: 5, b: 12}
{_id: 2, a: 15, c: "abc"}
{_id: 3, b: 99, c: "xyz"}
If a or b are missing, set to 0, if c is missing -> "unset"
Set Defaults
{_id: 1, a: 5, b: 12}
{_id: 2, a: 15, c: "abc"}
{_id: 3, b: 99, c: "xyz"}
If a or b are missing, set to 0, if c is missing -> "unset"
db.coll.update({}, [
{$replaceWith:{
}}
], {multi:true})
Set Defaults
{_id: 1, a: 5, b: 12}
{_id: 2, a: 15, c: "abc"}
{_id: 3, b: 99, c: "xyz"}
If a or b are missing, set to 0, if c is missing -> "unset"
db.coll.update({}, [
{$replaceWith:{$mergeObjects:[
]}}
], {multi:true})
Set Defaults
{_id: 1, a: 5, b: 12}
{_id: 2, a: 15, c: "abc"}
{_id: 3, b: 99, c: "xyz"}
If a or b are missing, set to 0, if c is missing -> "unset"
db.coll.update({}, [
{$replaceWith:{$mergeObjects:[
{ a:0, b:0, c:"unset" },
"$$ROOT"
]}}
], {multi:true})
Set Defaults
{_id: 1, a: 5, b: 12}
{_id: 2, a: 15, c: "abc"}
{_id: 3, b: 99, c: "xyz"}
If a or b are missing, set to 0, if c is missing -> "unset"
db.coll.update({}, [
{$replaceWith:{$mergeObjects:[
{ a:0, b:0, c:"unset" },
"$$ROOT"
]}}
], {multi:true})
Set Defaults
{_id: 1, a: 5, b: 12, c: "unset"}
{_id: 2, a: 15, b: 0, c: "abc"}
{_id: 3, a: 0, b: 99, c: "xyz"}
{ id: 1,
d: ISODate("2019-06-04T00:00:00"),
h: [
{ hour:"11", value: 296 },
{ hour:"12", value: 300 }
]}
id: X, d:Y, hour:Z, value: VAL
db.coll.update({id:X, d:Y},
[ {$set:{h:{$cond:{
if:
then:
else:
}}}}],
{upsert:true})
{ id: 1,
d: ISODate("2019-06-04T00:00:00"),
h: [
{ hour:"11", value: 296 },
{ hour:"12", value: 300 }
]}
id: X, d:Y, hour:Z, value: VAL
db.coll.update({id:X, d:Y},
[ {$set:{h:{$cond:{
if:
then:
else:
}}}}],
{upsert:true})
{ id: 1,
d: ISODate("2019-06-04T00:00:00"),
h: [
{ hour:"11", value: 296 },
{ hour:"12", value: 300 }
]}
id: X, d:Y, hour:Z, value: VAL
db.coll.update({id:X, d:Y},
[ {$set:{h:{$cond:{
if: {$in:[Z,{$ifNull:["$h.hour",[]]}]},
then:{$map:{
input:"$h",
in: {$cond:{ if:{$ne:["$$this.hour",Z]}, then:"$$this",
else: {hour: Z, value: {$sum:[ "$$this.value", VAL]}}
}}
}},
else:{$concatArrays:[{$ifNull:["$h",[]]},[{hour:Z,value:VAL}]]}
}}}}],
{upsert:true})
Recap:
Updates can be specified with aggregation pipeline
All fields from existing document can be accessed
Slightly slower, but a lot more powerful
THE FUTURE OF AGGREGATION
Better performance & optimizations
More stages & expressions
More options for output
Compass helper for aggregate
Unify different languages
THE FUTURE OF AGGREGATION
Better performance & optimizations
More stages & expressions
More options for output
Compass helper for aggregate
Unify different languages
THE FUTURE OF AGGREGATION
More options for output
More Options for Output
Prior to MongoDB 4.2
$out
coll
new_coll
$out
Prior to MongoDB 4.2
$out
coll
new_coll
$out
db.coll.aggregate( [ {pipeline}, ...
{$out: "new_coll"} ]);
Prior to MongoDB 4.2
$out
coll
new_coll
$out
db.coll.aggregate( [ {pipeline}, ...
{$out: "new_coll"} ]);
new_coll
○ must be unsharded
○ overwrites existing
New $merge stage
in aggregation pipeline
MongoDB 4.2
$merge
coll
coll2
$merge
MongoDB 4.2
$merge
db.coll.aggregate( [
{pipeline}, ...,
{$merge: { ... }
]);
coll
coll2
$merge
MongoDB 4.2
$merge
db.coll.aggregate( [
{pipeline}, ...,
{$merge: { ... }
]);
coll2
can exist
same or different 'db'
can be sharded
coll
coll2
$merge
coll
coll2
$merge
{ } { } { } { }
{ } { } { } { }
MongoDB 4.2
$merge syntax
{
$merge: {
into: <target>
}
}
$merge syntax
{
$merge: {
into: <target>
}
}
{$merge: "collection2"}
$merge syntax
{
$merge: {
into: <target>
}
}
{$merge: {db: "db2", coll: "collection2"}
$merge syntax
{
$merge: {
into: <target>
}
}
$merge syntax
{
$merge: {
into: <target>,
on: <fields>
}
}
on: "_id"
on: [ "_id", "shardkey(s)" ]
must be unique
$merge syntax
{
$merge: {
into: <target>,
on: <fields>
}
}
Actions
source target
Actions
nothing matched:
source target
Actions
nothing matched: usually insert
source target
Actions
nothing matched: usually insert
document matched:
source target
Actions
nothing matched: usually insert
document matched: overwrite? update? ???
source target
Actions
nothing matched: usually insert
document matched: update
source target
Actions
nothing matched: usually insert
document matched: update (merge)
source target
$merge syntax
{
$merge: {
into: <target>,
whenNotMatched:
whenMatched:
}
}
$merge syntax
{
$merge: {
into: <target>,
whenNotMatched:"insert",
whenMatched:
}
}
$merge syntax
{
$merge: {
into: <target>,
whenNotMatched:"insert",
whenMatched:"merge"
}
}
$merge syntax
{
$merge: {
into: <target>,
whenNotMatched:"insert"|"discard"|"fail",
whenMatched:"merge"
}
}
$merge syntax
{
$merge: {
into: <target>,
whenNotMatched:"insert"|"discard"|"fail",
whenMatched:"merge"|"replace"|"keepExisting"|"fail"|[...]
}
}
$merge syntax
{
$merge: {
into: <target>,
whenMatched:[...]
}
}
$merge syntax
{
$merge: {
into: <target>,
whenMatched:[<custom pipeline>]
}
}
$merge example
{
$merge: {
into: <target>,
whenMatched:[
{$addFields:{
}}
]
}
}
$merge example
{
$merge: {
into: <target>,
whenMatched:[
{$addFields:{
total:{$sum:["$total","$$new.total"]}
}}
]
}
}
$merge example
{
$merge: {
into: <target>,
whenMatched:[
{$set:{
total:{$sum:["$total","$$new.total"]}
}}
]
}
}
$merge example
{
$merge: {
into: <target>,
whenMatched:[
{$set:{
total:{$sum:["$total","$$new.total"]}
}}
]
}
}
$merge example
{
$merge: {
into: <target>,
whenMatched:[
{$set:{
total:{$sum:["$total","$$new.total"]}
}}
]
}
}
Incoming Target
{
_id: "37",
total: 64,
f1: "x"
}
{
_id: "37",
total: 245,
f1: "yyy"
}
Result:
{
}
$merge example
{
$merge: {
into: <target>,
whenMatched:[
{$set:{
total:{$sum:["$total","$$new.total"]}
}}
]
}
}
Incoming Target
{
_id: "37",
total: 64,
f1: "x"
}
{
_id: "37",
total: 245,
f1: "yyy"
}
Result:
{
_id: "37",
total: 309,
f1: "yyy"
}
$merge example 2
{
$merge: {
into: <target>,
whenMatched:[
{$replaceWith:{$mergeObjects:[
"$$new",
{total:{$sum:["$$new.total", "$total"]}}
]}}
]
}
}
$merge example 2
{
$merge: {
into: <target>,
whenMatched:[
{$replaceWith:{$mergeObjects:[
"$$new",
{total:{$sum:["$$new.total", "$total"]}}
]}}
]
}
}
Incoming Target
{
_id: "37",
total: 64,
f1: "x"
}
{
_id: "37",
total: 245,
f1: "yyy"
}
Result:
{
}
$merge example 2
{
$merge: {
into: <target>,
whenMatched:[
{$replaceWith:{$mergeObjects:[
"$$new",
{total:{$sum:["$$new.total", "$total"]}}
]}}
]
}
}
Incoming Target
{
_id: "37",
total: 64,
f1: "x"
}
{
_id: "37",
total: 245,
f1: "yyy"
}
Result:
{
_id: "37",
total: 309,
f1: "x"
}
$merge syntax
{
$merge: {
into: <target>,
whenMatched:[...]
}
}
$merge syntax
{
$merge: {
into: <target>,
let: { ... },
whenMatched:[ ...]
}
}
$merge syntax
{
$merge: {
into: <target>,
let: {new: "$$ROOT"},
whenMatched:[ ...]
}
}
{
$merge: {
into: <target>,
whenMatched:[
{$set:{
total:{$sum:["$total","$$new.total"]}
}}
]
}
}
{
$merge: {
into: <target>,
whenMatched:[
{$set:{
total:{$sum:["$total","$$new.total"]}
}}
]
}
}
{
$merge: {
into: <target>,
let: {itotal: "$total"},
whenMatched:[
{$set:{
total:{$sum:["$total","$$itotal"]}
}}
]
}
}
EXAMPLES
APPEND from TEMP collection
temp
real
data
real
Using $merge to append loaded and
cleansed records loaded into db
aggregate 'temp' and append valid records to 'data'
db.temp.aggregate( [
{ ... } /* pipeline to massage and cleanse data in temp */,
{$merge:{
into: "data",
whenMatched: "fail"
}}
]);
aggregate 'temp' and append valid records to 'data'
db.temp.aggregate( [
{ ... } /* pipeline to massage and cleanse data in temp */,
{$merge:{
into: "data",
whenMatched: "fail"
}}
]);
Similar to SQL's INSERT INTO T1 SELECT * from T2
EXAMPLES
Maintain Single View
mflix
users
users
mfriendbook
users
sv
Using $merge to populate/update
user fields from other services
mflix
users
users
mfriendbook
users
sv
Using $merge to populate/update
user fields from other services
sv.users
{
_id: "user253",
dob: ISODate(...),
f1: "yyy"
}
$merge updates fields from mflix.users collection into sv.users
collection. Our "_id" field is unique username
mflix_pipeline = [
{ "$project" : {
"_id" : "$username",
"mflix" : "$$ROOT"
}},
{ "$merge" : {
"into" : {
"db": "sv",
"collection" : "users"
},
"whenNotMatched" : "discard"
}}
]
(in mflix)
sv.users
{
_id: "user253",
dob: ISODate(...),
f1: "yyy"
}
$merge updates fields from mflix.users collection into sv.users
collection. Our "_id" field is unique username
mflix_pipeline = [
{ "$project" : {
"_id" : "$username",
"mflix" : "$$ROOT"
}},
{ "$merge" : {
"into" : {
"db": "sv",
"collection" : "users"
},
"whenNotMatched" : "discard"
}}
]
(in mflix) db.users.aggregate(mflix_pipeline)
sv.users
{
_id: "user253",
dob: ISODate(...),
f1: "yyy",
mflix: { ... }
}
$merge updates fields from mfriendbook.users collection into sv.users
collection. Our "_id" field is unique username
mfriendbook_pipeline = [
{ "$project" : {
"_id" : "$username",
"mfriendbook" : "$$ROOT"
}},
{ "$merge" : {
"into" : {
"db": "sv",
"collection" : "users"
},
"whenNotMatched" : "discard"
}}
]
(in mfriendbook)
sv.users
{
_id: "user253",
dob: ISODate(...),
f1: "yyy",
mflix: { ... }
}
$merge updates fields from mfriendbook.users collection into sv.users
collection. Our "_id" field is unique username
mfriendbook_pipeline = [
{ "$project" : {
"_id" : "$username",
"mfriendbook" : "$$ROOT"
}},
{ "$merge" : {
"into" : {
"db": "sv",
"collection" : "users"
},
"whenNotMatched" : "discard"
}}
]
(in mfriendbook) db.users.aggregate(mfriendbook_pipeline)
sv.users
{
_id: "user253",
dob: ISODate(...),
f1: "yyy",
mflix: { ... },
mfriendbook: { ... }
}
EXAMPLES
Populate ROLLUPS into summary table
registrations
real
regsummary
real
Using $merge to incrementally
update periodic rollups in summary
$merge to create/update periodic
rollups in summary collection (for all days)
db.regsummary.createIndex({event:1, date:1}, {unique: true});
$merge to create/update periodic
rollups in summary collection (for all days)
db.regsummary.createIndex({event:1, date:1}, {unique: true});
db.registrations.aggregate([
{$match: {event_id: "MDBW19"}},
{$group:{
_id:{$dateToString:{date:"$date",format:"%Y-%m-%d"}},
count: {$sum:1}
}},
{$project: {_id:0,event:"MDBW19",date:"$_id",total:"$count"}},
{$merge: {
into: "regsummary",
on: ["event", "date"]
}}
])
$merge to create/update periodic
rollups in summary collection (for all days)
db.regsummary.createIndex({event:1, date:1}, {unique: true});
db.registrations.aggregate([
{$match: {event_id: "MDBW19"}},
{$group:{
_id:{$dateToString:{date:"$date",format:"%Y-%m-%d"}},
count: {$sum:1}
}},
{$project: {_id:0,event:"MDBW19",date:"$_id",total:"$count"}},
{$merge: {
into: "regsummary",
on: ["event", "date"]
}}
])
{ "event" : "MDBW19", "date" : "2019-05-19", "total" : 33 }
{ "event" : "MDBW19", "date" : "2019-05-20", "total" : 15 }
{ "event" : "MDBW19", "date" : "2019-05-21", "total" : 24 }
$merge to incrementally update periodic rollups in summary
collection (for single day)
$merge to incrementally update periodic rollups in summary
collection (for single day)
db.registrations.aggregate([
{$match: {
event_id: "MDBW19",
date:{$gte:ISODate("2019-05-22"),$lt:ISODate("2019-05-23")}
}},
{$count: "total"},
{$addFields: {event:"MDBW19", "date":"2019-05-22"}},
{$merge: {
into: "regsummary",
on: ["event", "date"]
}}
])
$merge to incrementally update periodic rollups in summary
collection (for single day)
db.registrations.aggregate([
{$match: {
event_id: "MDBW19",
date:{$gte:ISODate("2019-05-22"),$lt:ISODate("2019-05-23")}
}},
{$count: "total"},
{$addFields: {event:"MDBW19", "date":"2019-05-22"}},
{$merge: {
into: "regsummary",
on: ["event", "date"]
}}
])
{ "event" : "MDBW19", "date" : "2019-05-19", "total" : 33 }
{ "event" : "MDBW19", "date" : "2019-05-20", "total" : 15 }
{ "event" : "MDBW19", "date" : "2019-05-21", "total" : 24 }
{ "event" : "MDBW19", "date" : "2019-05-22", "total" : 34 }
MongoDB World 2019: Aggregation Pipeline Power++: How MongoDB 4.2 Pipeline Empowers Queries, Updates, and Materialized Views [MongoDB]
https://github.com/asya999/mdbw17
Asya Kamsky
https://github.com/asya999/mdbw17
Regent
https://github.com/asya999/mdbw17
MongoDB World 2019: Aggregation Pipeline Power++: How MongoDB 4.2 Pipeline Empowers Queries, Updates, and Materialized Views [MongoDB]

More Related Content

What's hot (20)

PPTX
Aggregation in MongoDB
Kishor Parkhe
 
PPTX
How to leverage what's new in MongoDB 3.6
Maxime Beugnet
 
PPTX
Powerful Analysis with the Aggregation Pipeline
MongoDB
 
PDF
Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)
MongoSF
 
PPTX
Agg framework selectgroup feb2015 v2
MongoDB
 
PDF
Embedding a language into string interpolator
Michael Limansky
 
PDF
Mongodb Aggregation Pipeline
zahid-mian
 
PDF
MongoDB Europe 2016 - ETL for Pros – Getting Data Into MongoDB The Right Way
MongoDB
 
PDF
All Things Open 2016 -- Database Programming for Newbies
Dave Stokes
 
PDF
mobl - model-driven engineering lecture
zefhemel
 
PPTX
[MongoDB.local Bengaluru 2018] Tutorial: Pipeline Power - Doing More with Mon...
MongoDB
 
PPTX
ETL for Pros: Getting Data Into MongoDB
MongoDB
 
PPTX
MongoDB Analytics: Learn Aggregation by Example - Exploratory Analytics and V...
MongoDB
 
PDF
Mobile Web 5.0
Michael Galpin
 
PDF
CouchDB @ red dirt ruby conference
leinweber
 
PDF
PuppetCamp SEA @ Blk 71 - Nagios in under 10 mins with Puppet
Walter Heck
 
PDF
PuppetCamp SEA @ Blk 71 - Nagios in under 10 mins with Puppet
OlinData
 
KEY
PostgreSQLからMongoDBへ
Basuke Suzuki
 
PDF
Inside MongoDB: the Internals of an Open-Source Database
Mike Dirolf
 
PPT
DB2 Native XML
Amol Pujari
 
Aggregation in MongoDB
Kishor Parkhe
 
How to leverage what's new in MongoDB 3.6
Maxime Beugnet
 
Powerful Analysis with the Aggregation Pipeline
MongoDB
 
Map/reduce, geospatial indexing, and other cool features (Kristina Chodorow)
MongoSF
 
Agg framework selectgroup feb2015 v2
MongoDB
 
Embedding a language into string interpolator
Michael Limansky
 
Mongodb Aggregation Pipeline
zahid-mian
 
MongoDB Europe 2016 - ETL for Pros – Getting Data Into MongoDB The Right Way
MongoDB
 
All Things Open 2016 -- Database Programming for Newbies
Dave Stokes
 
mobl - model-driven engineering lecture
zefhemel
 
[MongoDB.local Bengaluru 2018] Tutorial: Pipeline Power - Doing More with Mon...
MongoDB
 
ETL for Pros: Getting Data Into MongoDB
MongoDB
 
MongoDB Analytics: Learn Aggregation by Example - Exploratory Analytics and V...
MongoDB
 
Mobile Web 5.0
Michael Galpin
 
CouchDB @ red dirt ruby conference
leinweber
 
PuppetCamp SEA @ Blk 71 - Nagios in under 10 mins with Puppet
Walter Heck
 
PuppetCamp SEA @ Blk 71 - Nagios in under 10 mins with Puppet
OlinData
 
PostgreSQLからMongoDBへ
Basuke Suzuki
 
Inside MongoDB: the Internals of an Open-Source Database
Mike Dirolf
 
DB2 Native XML
Amol Pujari
 

Similar to MongoDB World 2019: Aggregation Pipeline Power++: How MongoDB 4.2 Pipeline Empowers Queries, Updates, and Materialized Views [MongoDB] (20)

PDF
MongoDB .local Munich 2019: Aggregation Pipeline Power++: How MongoDB 4.2 Pip...
MongoDB
 
PDF
MongoDB .local Paris 2020: La puissance du Pipeline d'Agrégation de MongoDB
MongoDB
 
PDF
Aggregation Pipeline Power++: MongoDB 4.2 파이프 라인 쿼리, 업데이트 및 구체화된 뷰 소개 [MongoDB]
MongoDB
 
PPTX
The Aggregation Framework
MongoDB
 
PDF
Aggregation Framework MongoDB Days Munich
Norberto Leite
 
PPTX
The Aggregation Framework
MongoDB
 
PPTX
Aggregation Framework
MongoDB
 
PDF
Latinoware
kchodorow
 
PPTX
Query for json databases
Binh Le
 
PPTX
Introduction to MongoDB
Anton Fil
 
PDF
MongoDB
Hemant Kumar Tiwary
 
PPTX
Schema design with MongoDB (Dwight Merriman)
MongoSF
 
PDF
Full metal mongo
Israel Gutiérrez
 
PDF
MongoD Essentials
zahid-mian
 
KEY
MongoDB Aggregation Framework
Tyler Brock
 
PDF
MongoDB Aggregation Framework
Caserta
 
PPTX
Mongo db 101 dc group
John Ragan
 
PPTX
MongoDB 3.2 - Analytics
Massimo Brignoli
 
PDF
MongoDB
Rawin Windygallery
 
PPT
Schema design short
MongoDB
 
MongoDB .local Munich 2019: Aggregation Pipeline Power++: How MongoDB 4.2 Pip...
MongoDB
 
MongoDB .local Paris 2020: La puissance du Pipeline d'Agrégation de MongoDB
MongoDB
 
Aggregation Pipeline Power++: MongoDB 4.2 파이프 라인 쿼리, 업데이트 및 구체화된 뷰 소개 [MongoDB]
MongoDB
 
The Aggregation Framework
MongoDB
 
Aggregation Framework MongoDB Days Munich
Norberto Leite
 
The Aggregation Framework
MongoDB
 
Aggregation Framework
MongoDB
 
Latinoware
kchodorow
 
Query for json databases
Binh Le
 
Introduction to MongoDB
Anton Fil
 
Schema design with MongoDB (Dwight Merriman)
MongoSF
 
Full metal mongo
Israel Gutiérrez
 
MongoD Essentials
zahid-mian
 
MongoDB Aggregation Framework
Tyler Brock
 
MongoDB Aggregation Framework
Caserta
 
Mongo db 101 dc group
John Ragan
 
MongoDB 3.2 - Analytics
Massimo Brignoli
 
Schema design short
MongoDB
 
Ad

More from MongoDB (20)

PDF
MongoDB SoCal 2020: Migrate Anything* to MongoDB Atlas
MongoDB
 
PDF
MongoDB SoCal 2020: Go on a Data Safari with MongoDB Charts!
MongoDB
 
PDF
MongoDB SoCal 2020: Using MongoDB Services in Kubernetes: Any Platform, Devel...
MongoDB
 
PDF
MongoDB SoCal 2020: A Complete Methodology of Data Modeling for MongoDB
MongoDB
 
PDF
MongoDB SoCal 2020: From Pharmacist to Analyst: Leveraging MongoDB for Real-T...
MongoDB
 
PDF
MongoDB SoCal 2020: Best Practices for Working with IoT and Time-series Data
MongoDB
 
PDF
MongoDB SoCal 2020: MongoDB Atlas Jump Start
MongoDB
 
PDF
MongoDB .local San Francisco 2020: Powering the new age data demands [Infosys]
MongoDB
 
PDF
MongoDB .local San Francisco 2020: Using Client Side Encryption in MongoDB 4.2
MongoDB
 
PDF
MongoDB .local San Francisco 2020: Using MongoDB Services in Kubernetes: any ...
MongoDB
 
PDF
MongoDB .local San Francisco 2020: Go on a Data Safari with MongoDB Charts!
MongoDB
 
PDF
MongoDB .local San Francisco 2020: From SQL to NoSQL -- Changing Your Mindset
MongoDB
 
PDF
MongoDB .local San Francisco 2020: MongoDB Atlas Jumpstart
MongoDB
 
PDF
MongoDB .local San Francisco 2020: Tips and Tricks++ for Querying and Indexin...
MongoDB
 
PDF
MongoDB .local San Francisco 2020: A Complete Methodology of Data Modeling fo...
MongoDB
 
PDF
MongoDB .local San Francisco 2020: MongoDB Atlas Data Lake Technical Deep Dive
MongoDB
 
PDF
MongoDB .local San Francisco 2020: Developing Alexa Skills with MongoDB & Golang
MongoDB
 
PDF
MongoDB .local Paris 2020: Realm : l'ingrédient secret pour de meilleures app...
MongoDB
 
PDF
MongoDB .local Paris 2020: Upply @MongoDB : Upply : Quand le Machine Learning...
MongoDB
 
PDF
MongoDB .local Paris 2020: Les bonnes pratiques pour sécuriser MongoDB
MongoDB
 
MongoDB SoCal 2020: Migrate Anything* to MongoDB Atlas
MongoDB
 
MongoDB SoCal 2020: Go on a Data Safari with MongoDB Charts!
MongoDB
 
MongoDB SoCal 2020: Using MongoDB Services in Kubernetes: Any Platform, Devel...
MongoDB
 
MongoDB SoCal 2020: A Complete Methodology of Data Modeling for MongoDB
MongoDB
 
MongoDB SoCal 2020: From Pharmacist to Analyst: Leveraging MongoDB for Real-T...
MongoDB
 
MongoDB SoCal 2020: Best Practices for Working with IoT and Time-series Data
MongoDB
 
MongoDB SoCal 2020: MongoDB Atlas Jump Start
MongoDB
 
MongoDB .local San Francisco 2020: Powering the new age data demands [Infosys]
MongoDB
 
MongoDB .local San Francisco 2020: Using Client Side Encryption in MongoDB 4.2
MongoDB
 
MongoDB .local San Francisco 2020: Using MongoDB Services in Kubernetes: any ...
MongoDB
 
MongoDB .local San Francisco 2020: Go on a Data Safari with MongoDB Charts!
MongoDB
 
MongoDB .local San Francisco 2020: From SQL to NoSQL -- Changing Your Mindset
MongoDB
 
MongoDB .local San Francisco 2020: MongoDB Atlas Jumpstart
MongoDB
 
MongoDB .local San Francisco 2020: Tips and Tricks++ for Querying and Indexin...
MongoDB
 
MongoDB .local San Francisco 2020: A Complete Methodology of Data Modeling fo...
MongoDB
 
MongoDB .local San Francisco 2020: MongoDB Atlas Data Lake Technical Deep Dive
MongoDB
 
MongoDB .local San Francisco 2020: Developing Alexa Skills with MongoDB & Golang
MongoDB
 
MongoDB .local Paris 2020: Realm : l'ingrédient secret pour de meilleures app...
MongoDB
 
MongoDB .local Paris 2020: Upply @MongoDB : Upply : Quand le Machine Learning...
MongoDB
 
MongoDB .local Paris 2020: Les bonnes pratiques pour sécuriser MongoDB
MongoDB
 
Ad

Recently uploaded (20)

PDF
NASA A Researcher’s Guide to International Space Station : Physical Sciences ...
Dr. PANKAJ DHUSSA
 
PPTX
AI Penetration Testing Essentials: A Cybersecurity Guide for 2025
defencerabbit Team
 
PDF
“Computer Vision at Sea: Automated Fish Tracking for Sustainable Fishing,” a ...
Edge AI and Vision Alliance
 
PDF
The Rise of AI and IoT in Mobile App Tech.pdf
IMG Global Infotech
 
PPTX
New ThousandEyes Product Innovations: Cisco Live June 2025
ThousandEyes
 
PPTX
Mastering ODC + Okta Configuration - Chennai OSUG
HathiMaryA
 
PDF
Book industry state of the nation 2025 - Tech Forum 2025
BookNet Canada
 
PDF
Future-Proof or Fall Behind? 10 Tech Trends You Can’t Afford to Ignore in 2025
DIGITALCONFEX
 
PDF
SIZING YOUR AIR CONDITIONER---A PRACTICAL GUIDE.pdf
Muhammad Rizwan Akram
 
PPTX
Seamless Tech Experiences Showcasing Cross-Platform App Design.pptx
presentifyai
 
PDF
Newgen 2022-Forrester Newgen TEI_13 05 2022-The-Total-Economic-Impact-Newgen-...
darshakparmar
 
PDF
Peak of Data & AI Encore AI-Enhanced Workflows for the Real World
Safe Software
 
PPTX
Agentforce World Tour Toronto '25 - MCP with MuleSoft
Alexandra N. Martinez
 
PDF
CIFDAQ Market Wrap for the week of 4th July 2025
CIFDAQ
 
PPTX
Designing_the_Future_AI_Driven_Product_Experiences_Across_Devices.pptx
presentifyai
 
PDF
Staying Human in a Machine- Accelerated World
Catalin Jora
 
PPTX
Q2 FY26 Tableau User Group Leader Quarterly Call
lward7
 
PDF
UPDF - AI PDF Editor & Converter Key Features
DealFuel
 
DOCX
Python coding for beginners !! Start now!#
Rajni Bhardwaj Grover
 
PPTX
The Project Compass - GDG on Campus MSIT
dscmsitkol
 
NASA A Researcher’s Guide to International Space Station : Physical Sciences ...
Dr. PANKAJ DHUSSA
 
AI Penetration Testing Essentials: A Cybersecurity Guide for 2025
defencerabbit Team
 
“Computer Vision at Sea: Automated Fish Tracking for Sustainable Fishing,” a ...
Edge AI and Vision Alliance
 
The Rise of AI and IoT in Mobile App Tech.pdf
IMG Global Infotech
 
New ThousandEyes Product Innovations: Cisco Live June 2025
ThousandEyes
 
Mastering ODC + Okta Configuration - Chennai OSUG
HathiMaryA
 
Book industry state of the nation 2025 - Tech Forum 2025
BookNet Canada
 
Future-Proof or Fall Behind? 10 Tech Trends You Can’t Afford to Ignore in 2025
DIGITALCONFEX
 
SIZING YOUR AIR CONDITIONER---A PRACTICAL GUIDE.pdf
Muhammad Rizwan Akram
 
Seamless Tech Experiences Showcasing Cross-Platform App Design.pptx
presentifyai
 
Newgen 2022-Forrester Newgen TEI_13 05 2022-The-Total-Economic-Impact-Newgen-...
darshakparmar
 
Peak of Data & AI Encore AI-Enhanced Workflows for the Real World
Safe Software
 
Agentforce World Tour Toronto '25 - MCP with MuleSoft
Alexandra N. Martinez
 
CIFDAQ Market Wrap for the week of 4th July 2025
CIFDAQ
 
Designing_the_Future_AI_Driven_Product_Experiences_Across_Devices.pptx
presentifyai
 
Staying Human in a Machine- Accelerated World
Catalin Jora
 
Q2 FY26 Tableau User Group Leader Quarterly Call
lward7
 
UPDF - AI PDF Editor & Converter Key Features
DealFuel
 
Python coding for beginners !! Start now!#
Rajni Bhardwaj Grover
 
The Project Compass - GDG on Campus MSIT
dscmsitkol
 

MongoDB World 2019: Aggregation Pipeline Power++: How MongoDB 4.2 Pipeline Empowers Queries, Updates, and Materialized Views [MongoDB]

  • 1. asya999 AGGREGATION How MongoDB 4.2 Pipeline is Powering Queries, Updates and Views PIPELINE POWER+++
  • 2. asya999 AGGREGATION How MongoDB 4.2 Pipeline is Powering Queries, Updates and Views PIPELINE POWER+++
  • 4. #MDBW17 Analytics with MongoDB Aggregation Framework @asya999 by Asya Kamsky, Lead MongoDB Maven PIPELINE POWER
  • 6. ps ax | grep mongod | head 1 *nix command line pipe PIPELINE
  • 7. $match $group | $sort| Input stream {} {} {} {} Result {} {} ... PIPELINE MongoDB document pipeline
  • 9. Stage 1 Stage 2 Stage 3 Stage 4 {} {} {} {} {} {} {} {} DATA PIPELINE {} {} {} {} {"$stage":{ ... }} START Collection View Special stage STAGES
  • 10. {title: "The Great Gatsby", language: "English", subjects: "Long Island"} {title: "The Great Gatsby", language: "English", subjects: "New York"} {title: "The Great Gatsby", language: "English", subjects: "1920s"} {title: "The Great Gatsby", language: "English", subjects: [ "Long Island", "New York", "1920s"] }, {"$match":{"language":"English"}} $match { _id:"Long Island", count: 1 }, $group { _id: "New York", count: 2 }, $unwind { _id: "1920s", count: 1 }, $sort $skip$limit $project {"$unwind":"$subjects"} {"$group":{"_id":"$subjects", "count":{"$sum:1}} { _id: "Harlem", count: 1 }, { _id: "Long Island", count: 1 }, { _id: "New York", count: 2 }, { _id: "1920s", count: 1 }, {title: "Open City", language: "English", subjects: [ "New York" "Harlem" ] } { title: "The Great Gatsby", language: "English", subjects: [ "Long Island", "New York", "1920s"] }, { title: "War and Peace", language: "Russian", subjects: [ "Russia", "War of 1812", "Napoleon"] }, { title: "Open City", language: "English", subjects: [ "New York", "Harlem" ] }, {title: "Open City", language: "English", subjects: "New York"} {title: "Open City", language: "English", subjects: "Harlem"} { _id: "Harlem", count: 1 }, {"$sort:{"count":-1} {"$limit":3} {"$project":...}
  • 13. {title: "The Great Gatsby", language: "English". subjects: "Long Island"} {title: "The Great Gatsby", language: "English", subjects: "New York"} {title: "The Great Gatsby", language: "English", subjects: "1920s"} {title: "The Great Gatsby", language: "English", subjects: [ "Long Island", "New York", "1920s"] }, {"$match":{"language":"English"}} $match { _id:"Long Island", count: 1 }, $group { _id: "New York", count: 2 }, $unwind { _id: "1920s", count: 1 }, $sort $skip$limit $project {"$unwind":"$subjects"} {"$group":{"_id":"$subjects", "count":{"$sum:1}} { _id: "Harlem", count: 1 }, { _id: "Long Island", count: 1 }, { _id: "New York", count: 2 }, { _id: "1920s", count: 1 }, {title: "Open City", language: "English", subjects: [ "New York" "Harlem" ] } { title: "The Great Gatsby", language: "English", subjects: [ "Long Island", "New York", "1920s"] }, { title: "War and Peace", language: "Russian", subjects: [ "Russia", "War of 1812", "Napoleon"] }, { title: "Open City", language: "English", subjects: [ "New York", "Harlem" ] }, {title: "Open City", language: "English", subjects: "New York"} {title: "Open City", language: "English", subjects: "Harlem"} { _id: "Harlem", count: 1 }, {"$sort:{"count":-1} {"$limit":3} {"$project":...} $group $sort 1
  • 14. INPUT STAGE RESULTSSTAGE STREAMING RESOURCE USE Each document is streamed through in RAM
  • 15. INPUT STAGE RESULTSSTAGE BLOCKING RESOURCE USE Everything has to be kept in RAM (or spill)
  • 21. $sort$match $group start=ISODate("...") end=ISODate("...") { user: "303900", ipaddr: "71.56.112.56", ts:ISODate("2017-05-08T...") } {$match:{ts:{$gte:start,$lt:end}}}, {$sort:{ts:1}}, {$group:{_id:"$user",ips:{$push:{ip:"$ipaddr", ts:"$ts"}}, diffIps:{$addToSet:"$ipaddr"}}}, {$match:{"diffIps.1":{$exists:true}}}, {$addFields:{diffs: {$filter:{ input:{$map:{ input: {$range:[0,{$subtract:[{$size:"$ips"},1]}]}, as:"i", in:{$let:{vars:{ip1:{$arrayElemAt:["$ips","$$i"]}, ip2:{$arrayElemAt:["$ips",{$add:["$$i",1]}]}}, in:{ diff:{$cond:{ if:{$ne:["$$ip1.ip","$$ip2.ip"]}, then:{$divide:[{$subtract:["$$ip2.ts","$$ip1.ts"]},60000]}, else: 9999 }}, ip1:"$$ip1.ip", t1:"$$ip1.ts", ip2:"$$ip2.ip", t2:"$$ip2.ts" }}}}}, cond:{$lt:["$$this.diff",10]} }}}}, {$match:{"diffs":{$ne:[]}}}, {$project:{_id:0, user:"$_id", suspectLogins:"$diffs"}} { "user" : "35237073", "suspectLogins" : [ {"diff": 4.8333333333, "ip1": "106.220.151.16", "t1":"2017-05-08T06:58", "ip2": "223.182.113.15" "t2":"2017-05-08T07:03" }, {"diff": 8.3, "ip1": "223.182.113.15", "t1":"2017-05-08T07:03", "ip2": "49.206.217.26", "t2":"2017-05-08T07:11" } ] } $match $addFields $match $project
  • 24. #MDBW17 Analytics with MongoDB Aggregation Framework @asya999 by Asya Kamsky, Lead MongoDB Maven PIPELINE POWER https://github.com/asya999/mdbw17
  • 28. THE FUTURE OF AGGREGATION Better performance & optimizations More stages & expressions More options for output Compass helper for aggregate Unify different languages
  • 29. THE FUTURE OF AGGREGATION Better performance & optimizations More stages & expressions More options for output Compass helper for aggregate Unify different languages
  • 30. THE FUTURE OF AGGREGATION Better performance & optimizations More stages & expressions More options for output Compass helper for aggregate Unify different languages
  • 31. THE FUTURE OF AGGREGATION More options for output Unify different languages
  • 32. More options for output Unify different languages THE PRESENT OF AGGREGATION
  • 34. {children: [ {name:"Max", dob:"1994-12-01", dep:true}, {name:"Sam", dob:"1997-09-28", dep:true}, {name:"Kim", dob:"2000-02-29", dep:true} ]} AGGREGATION Unify Different Languages
  • 35. {children: [ {name:"Max", dob:"1994-12-01", dep:true}, {name:"Sam", dob:"1997-09-28", dep:true}, {name:"Kim", dob:"2000-02-29", dep:true} ]} AGGREGATION Unify Different Languages db.c.aggregate([ {$addFields:{ numChildren:{$size:"$children"}, numDependents:{$size:{ $filter:{ input:"$children.dep", cond: "$$this" } }} }}, ... ])
  • 36. {children: [ {name:"Max", dob:"1994-12-01", dep:true}, {name:"Sam", dob:"1997-09-28", dep:true}, {name:"Kim", dob:"2000-02-29", dep:true} ]} AGGREGATION FIND Unify Different Languages db.c.aggregate([ {$addFields:{ numChildren:{$size:"$children"}, numDependents:{$size:{ $filter:{ input:"$children.dep", cond: "$$this" } }} }}, ... ])
  • 37. {children: [ {name:"Max", dob:"1994-12-01", dep:true}, {name:"Sam", dob:"1997-09-28", dep:true}, {name:"Kim", dob:"2000-02-29", dep:true} ]} AGGREGATION FIND Unify Different Languages db.c.find ( {$expr:{ $lt:[ {$size:{$filter:{ input: "$children.dep", cond: "$$this" }}}, 2 ] }} )
  • 38. {children: [ {name:"Max", dob:"1994-12-01", dep:true}, {name:"Sam", dob:"1997-09-28", dep:true}, {name:"Kim", dob:"2000-02-29", dep:true} ]} AGGREGATION FIND UPDATE Unify Different Languages db.c.find ( {$expr:{ $lt:[ {$size:{$filter:{ input: "$children.dep", cond: "$$this" }}}, 2 ] }} )
  • 39. {children: [ {name:"Max", dob:"1994-12-01", dep:true}, {name:"Sam", dob:"1997-09-28", dep:true}, {name:"Kim", dob:"2000-02-29", dep:true} ]} AGGREGATION FIND UPDATE Unify Different Languages db.c.update( {$expr:{ $anyElementTrue:{$map:{ input:"$children", in: {$and:[ {$lt:["$$this.dob","1997-01-22"]}, "$$this.dep" ]} }} }}, {$set:{ audit:true }} )
  • 43. <update> { $set: { }, $inc: { }, $... } Update { f1: <value>, f2: <value>, ... }
  • 44. <update> { } OR [ ] Update in 4.2 [ ]
  • 47. { $addFields: { } } { $project: { } } { $replaceRoot: { } } { $set: { } } { $unset: [ ] } { $replaceWith: { } }
  • 48. { _id: 1 } { _id: 1, a: 10 } { _id: 1, a: 100 } --- { _id: 1, a: "10" } db.coll.update({_id:1}, {$inc:{a:1}}, {upsert:true}) { _id: 1, a: 1 } { _id: 1, a: 11 } { _id: 1, a: 101 } { _id: 1, a: 1 } "errmsg" : "Cannot apply to a value of non-numeric type."
  • 49. { _id: 1 } { _id: 1, a: 10 } { _id: 1, a: 100 } --- { _id: 1, a: "10" } db.coll.update({_id:1}, [ {$set:{a:{$sum:["$a",1]}}} ], {upsert:true}) { _id: 1, a: 1 } { _id: 1, a: 11 } { _id: 1, a: 101 } { _id: 1, a: 1 } { _id: 1, a: 1 }
  • 50. { _id: 1 } { _id: 1, a: 10 } { _id: 1, a: 100 } --- { _id: 1, a: "10" } db.coll.update({_id:1}, [ {$set:{a:{$add:["$a",1]}}} ], {upsert:true}) { _id: 1, a: 1 } { _id: 1, a: 11 } { _id: 1, a: 101 } { _id: 1, a: 1 } "errmsg" : "$add only supports numeric or date types, not string"
  • 51. { _id: 1 } { _id: 1, a: 10 } { _id: 1, a: 100 } --- { _id: 1, a: "10" } db.coll.update({_id:1}, [ ], {upsert:true}) { _id:1, a: 21 } { _id: 1, a: 11 } { _id: 1, a: 101 } { _id:1, a: 21 }
  • 52. { _id: 1 } { _id: 1, a: 10 } { _id: 1, a: 100 } --- { _id: 1, a: "10" } db.coll.update({_id:1}, [ {$set:{a:{$cond:{ if: then: , else: }} }}] {upsert:true}) { _id:1, a: 21 } { _id: 1, a: 11 } { _id: 1, a: 101 } { _id:1, a: 21 }
  • 53. { _id: 1 } { _id: 1, a: 10 } { _id: 1, a: 100 } --- { _id: 1, a: "10" } db.coll.update({_id:1}, [ {$set:{a:{$cond:{ if: {$eq:[{$type:"$a"},"missing"]}, then: , else: }} }}], {upsert:true}) { _id:1, a: 21 } { _id: 1, a: 11 } { _id: 1, a: 101 } { _id:1, a: 21 }
  • 54. { _id: 1 } { _id: 1, a: 10 } { _id: 1, a: 100 } --- { _id: 1, a: "10" } db.coll.update({_id:1}, [ {$set:{a:{$cond:{ if: {$eq:[{$type:"$a"},"missing"]}, then: 21, else: {$sum:["$a", 1]} }} }}], {upsert:true}) { _id:1, a: 21 } { _id: 1, a: 11 } { _id: 1, a: 101 } { _id:1, a: 21 }
  • 55. { _id: 1 } { _id: 1, a: 10 } { _id: 1, a: 100 } --- { _id: 1, a: "10" } db.coll.update({_id:1}, [ {$set:{a:{$cond:{ if: {$eq:[{$type:"$a"},"missing"]}, then: 21, else: {$sum:["$a", 1]} }} }}], {upsert:true}) { _id:1, a: 21 } { _id: 1, a: 11 } { _id: 1, a: 100 } { _id:1, a: 21 }
  • 56. { _id: 1 } { _id: 1, a: 10 } { _id: 1, a: 100 } --- { _id: 1, a: "10" } db.coll.update({_id:1}, [ {$set:{a:{$min:[ 100, {$cond:{ if: {$eq:[{$type:"$a"},"missing"]}, then: 21, else: {$sum:["$a", 1]} }}]} }}], {upsert:true}) { _id:1, a: 21 } { _id: 1, a: 11 } { _id: 1, a: 100 } { _id:1, a: 21 }
  • 57. { _id: 1 } { _id: 1, a: 10 } { _id: 1, a: 100 } --- { _id: 1, a: "10" } db.coll.update({_id:1}, [ {$set:{a:{$min:[ 100, {$cond:{ if: {$eq:[{$type:"$a"},"missing"]}, then: 21, else: {$sum:["$a", 1]} }}]} }}], {upsert:true}) { _id:1, a: 21 } { _id: 1, a: 11 } { _id: 1, a: 100 } { _id:1, a: 21 } { _id:1, a: 1 }
  • 58. { _id: 1 } { _id: 1, a: 10 } { _id: 1, a: 100 } --- { _id: 1, a: "10" } db.coll.update({_id:1}, [ {$set:{a:{$min:[ 100, {$cond:{ if: {$eq:[{$type:"$a"},"missing"]}, then: 21, else: {$sum:["$a", 1]} }}]}, prev_a: "$a"}}], {upsert:true}) { _id:1, a: 21 } { _id: 1, a: 11 } { _id: 1, a: 100 } { _id:1, a: 21 } { _id:1, a: 1 }
  • 59. { _id: 1 } { _id: 1, a: 10 } { _id: 1, a: 100 } --- { _id: 1, a: "10" } db.coll.update({_id:1}, [ {$set:{a:{$min:[ 100, {$cond:{ if: {$eq:[{$type:"$a"},"missing"]}, then: 21, else: {$sum:["$a", 1]} }}]}, prev_a: "$a"}}], {upsert:true}) { _id:1, a: 21 } { _id: 1, a: 11, prev_a: 10 } { _id: 1, a: 100, prev_a: 100 } { _id:1, a: 21 } { _id:1, a: 1, prev_a: "10" }
  • 61. {_id: 1, a: 5, b: 12} {_id: 2, a: 15, c: "abc"} {_id: 3, b: 99, c: "xyz"} If a or b are missing, set to 0, if c is missing -> "unset" Set Defaults
  • 62. {_id: 1, a: 5, b: 12} {_id: 2, a: 15, c: "abc"} {_id: 3, b: 99, c: "xyz"} If a or b are missing, set to 0, if c is missing -> "unset" db.coll.update({}, [ {$replaceWith:{ }} ], {multi:true}) Set Defaults
  • 63. {_id: 1, a: 5, b: 12} {_id: 2, a: 15, c: "abc"} {_id: 3, b: 99, c: "xyz"} If a or b are missing, set to 0, if c is missing -> "unset" db.coll.update({}, [ {$replaceWith:{$mergeObjects:[ ]}} ], {multi:true}) Set Defaults
  • 64. {_id: 1, a: 5, b: 12} {_id: 2, a: 15, c: "abc"} {_id: 3, b: 99, c: "xyz"} If a or b are missing, set to 0, if c is missing -> "unset" db.coll.update({}, [ {$replaceWith:{$mergeObjects:[ { a:0, b:0, c:"unset" }, "$$ROOT" ]}} ], {multi:true}) Set Defaults
  • 65. {_id: 1, a: 5, b: 12} {_id: 2, a: 15, c: "abc"} {_id: 3, b: 99, c: "xyz"} If a or b are missing, set to 0, if c is missing -> "unset" db.coll.update({}, [ {$replaceWith:{$mergeObjects:[ { a:0, b:0, c:"unset" }, "$$ROOT" ]}} ], {multi:true}) Set Defaults {_id: 1, a: 5, b: 12, c: "unset"} {_id: 2, a: 15, b: 0, c: "abc"} {_id: 3, a: 0, b: 99, c: "xyz"}
  • 66. { id: 1, d: ISODate("2019-06-04T00:00:00"), h: [ { hour:"11", value: 296 }, { hour:"12", value: 300 } ]} id: X, d:Y, hour:Z, value: VAL db.coll.update({id:X, d:Y}, [ {$set:{h:{$cond:{ if: then: else: }}}}], {upsert:true})
  • 67. { id: 1, d: ISODate("2019-06-04T00:00:00"), h: [ { hour:"11", value: 296 }, { hour:"12", value: 300 } ]} id: X, d:Y, hour:Z, value: VAL db.coll.update({id:X, d:Y}, [ {$set:{h:{$cond:{ if: then: else: }}}}], {upsert:true}) { id: 1, d: ISODate("2019-06-04T00:00:00"), h: [ { hour:"11", value: 296 }, { hour:"12", value: 300 } ]} id: X, d:Y, hour:Z, value: VAL db.coll.update({id:X, d:Y}, [ {$set:{h:{$cond:{ if: {$in:[Z,{$ifNull:["$h.hour",[]]}]}, then:{$map:{ input:"$h", in: {$cond:{ if:{$ne:["$$this.hour",Z]}, then:"$$this", else: {hour: Z, value: {$sum:[ "$$this.value", VAL]}} }} }}, else:{$concatArrays:[{$ifNull:["$h",[]]},[{hour:Z,value:VAL}]]} }}}}], {upsert:true})
  • 68. Recap: Updates can be specified with aggregation pipeline All fields from existing document can be accessed Slightly slower, but a lot more powerful
  • 69. THE FUTURE OF AGGREGATION Better performance & optimizations More stages & expressions More options for output Compass helper for aggregate Unify different languages
  • 70. THE FUTURE OF AGGREGATION Better performance & optimizations More stages & expressions More options for output Compass helper for aggregate Unify different languages
  • 71. THE FUTURE OF AGGREGATION More options for output
  • 73. Prior to MongoDB 4.2 $out coll new_coll $out
  • 74. Prior to MongoDB 4.2 $out coll new_coll $out db.coll.aggregate( [ {pipeline}, ... {$out: "new_coll"} ]);
  • 75. Prior to MongoDB 4.2 $out coll new_coll $out db.coll.aggregate( [ {pipeline}, ... {$out: "new_coll"} ]); new_coll ○ must be unsharded ○ overwrites existing
  • 76. New $merge stage in aggregation pipeline
  • 78. MongoDB 4.2 $merge db.coll.aggregate( [ {pipeline}, ..., {$merge: { ... } ]); coll coll2 $merge
  • 79. MongoDB 4.2 $merge db.coll.aggregate( [ {pipeline}, ..., {$merge: { ... } ]); coll2 can exist same or different 'db' can be sharded coll coll2 $merge
  • 80. coll coll2 $merge { } { } { } { } { } { } { } { } MongoDB 4.2
  • 82. $merge syntax { $merge: { into: <target> } } {$merge: "collection2"}
  • 83. $merge syntax { $merge: { into: <target> } } {$merge: {db: "db2", coll: "collection2"}
  • 85. $merge syntax { $merge: { into: <target>, on: <fields> } } on: "_id" on: [ "_id", "shardkey(s)" ] must be unique
  • 86. $merge syntax { $merge: { into: <target>, on: <fields> } }
  • 89. Actions nothing matched: usually insert source target
  • 90. Actions nothing matched: usually insert document matched: source target
  • 91. Actions nothing matched: usually insert document matched: overwrite? update? ??? source target
  • 92. Actions nothing matched: usually insert document matched: update source target
  • 93. Actions nothing matched: usually insert document matched: update (merge) source target
  • 94. $merge syntax { $merge: { into: <target>, whenNotMatched: whenMatched: } }
  • 95. $merge syntax { $merge: { into: <target>, whenNotMatched:"insert", whenMatched: } }
  • 96. $merge syntax { $merge: { into: <target>, whenNotMatched:"insert", whenMatched:"merge" } }
  • 97. $merge syntax { $merge: { into: <target>, whenNotMatched:"insert"|"discard"|"fail", whenMatched:"merge" } }
  • 98. $merge syntax { $merge: { into: <target>, whenNotMatched:"insert"|"discard"|"fail", whenMatched:"merge"|"replace"|"keepExisting"|"fail"|[...] } }
  • 99. $merge syntax { $merge: { into: <target>, whenMatched:[...] } }
  • 100. $merge syntax { $merge: { into: <target>, whenMatched:[<custom pipeline>] } }
  • 101. $merge example { $merge: { into: <target>, whenMatched:[ {$addFields:{ }} ] } }
  • 102. $merge example { $merge: { into: <target>, whenMatched:[ {$addFields:{ total:{$sum:["$total","$$new.total"]} }} ] } }
  • 103. $merge example { $merge: { into: <target>, whenMatched:[ {$set:{ total:{$sum:["$total","$$new.total"]} }} ] } }
  • 104. $merge example { $merge: { into: <target>, whenMatched:[ {$set:{ total:{$sum:["$total","$$new.total"]} }} ] } }
  • 105. $merge example { $merge: { into: <target>, whenMatched:[ {$set:{ total:{$sum:["$total","$$new.total"]} }} ] } } Incoming Target { _id: "37", total: 64, f1: "x" } { _id: "37", total: 245, f1: "yyy" } Result: { }
  • 106. $merge example { $merge: { into: <target>, whenMatched:[ {$set:{ total:{$sum:["$total","$$new.total"]} }} ] } } Incoming Target { _id: "37", total: 64, f1: "x" } { _id: "37", total: 245, f1: "yyy" } Result: { _id: "37", total: 309, f1: "yyy" }
  • 107. $merge example 2 { $merge: { into: <target>, whenMatched:[ {$replaceWith:{$mergeObjects:[ "$$new", {total:{$sum:["$$new.total", "$total"]}} ]}} ] } }
  • 108. $merge example 2 { $merge: { into: <target>, whenMatched:[ {$replaceWith:{$mergeObjects:[ "$$new", {total:{$sum:["$$new.total", "$total"]}} ]}} ] } } Incoming Target { _id: "37", total: 64, f1: "x" } { _id: "37", total: 245, f1: "yyy" } Result: { }
  • 109. $merge example 2 { $merge: { into: <target>, whenMatched:[ {$replaceWith:{$mergeObjects:[ "$$new", {total:{$sum:["$$new.total", "$total"]}} ]}} ] } } Incoming Target { _id: "37", total: 64, f1: "x" } { _id: "37", total: 245, f1: "yyy" } Result: { _id: "37", total: 309, f1: "x" }
  • 110. $merge syntax { $merge: { into: <target>, whenMatched:[...] } }
  • 111. $merge syntax { $merge: { into: <target>, let: { ... }, whenMatched:[ ...] } }
  • 112. $merge syntax { $merge: { into: <target>, let: {new: "$$ROOT"}, whenMatched:[ ...] } }
  • 114. { $merge: { into: <target>, whenMatched:[ {$set:{ total:{$sum:["$total","$$new.total"]} }} ] } } { $merge: { into: <target>, let: {itotal: "$total"}, whenMatched:[ {$set:{ total:{$sum:["$total","$$itotal"]} }} ] } }
  • 116. temp real data real Using $merge to append loaded and cleansed records loaded into db
  • 117. aggregate 'temp' and append valid records to 'data' db.temp.aggregate( [ { ... } /* pipeline to massage and cleanse data in temp */, {$merge:{ into: "data", whenMatched: "fail" }} ]);
  • 118. aggregate 'temp' and append valid records to 'data' db.temp.aggregate( [ { ... } /* pipeline to massage and cleanse data in temp */, {$merge:{ into: "data", whenMatched: "fail" }} ]); Similar to SQL's INSERT INTO T1 SELECT * from T2
  • 120. mflix users users mfriendbook users sv Using $merge to populate/update user fields from other services
  • 121. mflix users users mfriendbook users sv Using $merge to populate/update user fields from other services sv.users { _id: "user253", dob: ISODate(...), f1: "yyy" }
  • 122. $merge updates fields from mflix.users collection into sv.users collection. Our "_id" field is unique username mflix_pipeline = [ { "$project" : { "_id" : "$username", "mflix" : "$$ROOT" }}, { "$merge" : { "into" : { "db": "sv", "collection" : "users" }, "whenNotMatched" : "discard" }} ] (in mflix) sv.users { _id: "user253", dob: ISODate(...), f1: "yyy" }
  • 123. $merge updates fields from mflix.users collection into sv.users collection. Our "_id" field is unique username mflix_pipeline = [ { "$project" : { "_id" : "$username", "mflix" : "$$ROOT" }}, { "$merge" : { "into" : { "db": "sv", "collection" : "users" }, "whenNotMatched" : "discard" }} ] (in mflix) db.users.aggregate(mflix_pipeline) sv.users { _id: "user253", dob: ISODate(...), f1: "yyy", mflix: { ... } }
  • 124. $merge updates fields from mfriendbook.users collection into sv.users collection. Our "_id" field is unique username mfriendbook_pipeline = [ { "$project" : { "_id" : "$username", "mfriendbook" : "$$ROOT" }}, { "$merge" : { "into" : { "db": "sv", "collection" : "users" }, "whenNotMatched" : "discard" }} ] (in mfriendbook) sv.users { _id: "user253", dob: ISODate(...), f1: "yyy", mflix: { ... } }
  • 125. $merge updates fields from mfriendbook.users collection into sv.users collection. Our "_id" field is unique username mfriendbook_pipeline = [ { "$project" : { "_id" : "$username", "mfriendbook" : "$$ROOT" }}, { "$merge" : { "into" : { "db": "sv", "collection" : "users" }, "whenNotMatched" : "discard" }} ] (in mfriendbook) db.users.aggregate(mfriendbook_pipeline) sv.users { _id: "user253", dob: ISODate(...), f1: "yyy", mflix: { ... }, mfriendbook: { ... } }
  • 127. registrations real regsummary real Using $merge to incrementally update periodic rollups in summary
  • 128. $merge to create/update periodic rollups in summary collection (for all days) db.regsummary.createIndex({event:1, date:1}, {unique: true});
  • 129. $merge to create/update periodic rollups in summary collection (for all days) db.regsummary.createIndex({event:1, date:1}, {unique: true}); db.registrations.aggregate([ {$match: {event_id: "MDBW19"}}, {$group:{ _id:{$dateToString:{date:"$date",format:"%Y-%m-%d"}}, count: {$sum:1} }}, {$project: {_id:0,event:"MDBW19",date:"$_id",total:"$count"}}, {$merge: { into: "regsummary", on: ["event", "date"] }} ])
  • 130. $merge to create/update periodic rollups in summary collection (for all days) db.regsummary.createIndex({event:1, date:1}, {unique: true}); db.registrations.aggregate([ {$match: {event_id: "MDBW19"}}, {$group:{ _id:{$dateToString:{date:"$date",format:"%Y-%m-%d"}}, count: {$sum:1} }}, {$project: {_id:0,event:"MDBW19",date:"$_id",total:"$count"}}, {$merge: { into: "regsummary", on: ["event", "date"] }} ]) { "event" : "MDBW19", "date" : "2019-05-19", "total" : 33 } { "event" : "MDBW19", "date" : "2019-05-20", "total" : 15 } { "event" : "MDBW19", "date" : "2019-05-21", "total" : 24 }
  • 131. $merge to incrementally update periodic rollups in summary collection (for single day)
  • 132. $merge to incrementally update periodic rollups in summary collection (for single day) db.registrations.aggregate([ {$match: { event_id: "MDBW19", date:{$gte:ISODate("2019-05-22"),$lt:ISODate("2019-05-23")} }}, {$count: "total"}, {$addFields: {event:"MDBW19", "date":"2019-05-22"}}, {$merge: { into: "regsummary", on: ["event", "date"] }} ])
  • 133. $merge to incrementally update periodic rollups in summary collection (for single day) db.registrations.aggregate([ {$match: { event_id: "MDBW19", date:{$gte:ISODate("2019-05-22"),$lt:ISODate("2019-05-23")} }}, {$count: "total"}, {$addFields: {event:"MDBW19", "date":"2019-05-22"}}, {$merge: { into: "regsummary", on: ["event", "date"] }} ]) { "event" : "MDBW19", "date" : "2019-05-19", "total" : 33 } { "event" : "MDBW19", "date" : "2019-05-20", "total" : 15 } { "event" : "MDBW19", "date" : "2019-05-21", "total" : 24 } { "event" : "MDBW19", "date" : "2019-05-22", "total" : 34 }