Using Change Streams to Keep Up with Your Data
2 likes5,174 views
The document discusses the use of change streams in MongoDB to monitor real-time data changes in collections, explaining their characteristics and functionalities. Key features include operation types like insert, delete, update, and the ability to resume after interruptions while utilizing the aggregation framework for filtering changes. It emphasizes the importance of collection access controls and total ordering of changes across shards for data integrity and performance.
![# M D B l o c a l
SP
CHANGE STREAMS ARE RESUMABLE
P
S
{_id: <resumeToken>,
operationType: ‘update’,
ns: {db: ‘data’, coll: ‘temperature’},
documentKey: {_id: ‘room12345’},
updateDescription: {updatedFields: {temperature: 70},
removedFields: [‘humidity’]}
}
var newChangeStream = coll.watch({ resumeAfter: <cachedResumeToken> });](https://image.slidesharecdn.com/1200-1240-using-change-strea-c3bf1aea-cfd8-4491-870e-0aab2523c7fa-2139307395-171018133356/85/Using-Change-Streams-to-Keep-Up-with-Your-Data-11-320.jpg)
![# M D B l o c a l
CHANGE STREAMS UTILIZE THE POWER
OF THE AGGREGATION FRAMEWORK
$match $project $addFields $replaceRoot $redact
var changeStream = coll.watch([{ $match: { $or: [ { operationType: 'delete' }, { operationType: 'replace' } ] });](https://image.slidesharecdn.com/1200-1240-using-change-strea-c3bf1aea-cfd8-4491-870e-0aab2523c7fa-2139307395-171018133356/85/Using-Change-Streams-to-Keep-Up-with-Your-Data-13-320.jpg)
![# M D B l o c a l
5 OPERATION TYPES
{ _id: (resumeToken),
operationType: ‘insert’,
ns: {db: ‘test’, coll: ‘foo’},
documentKey: {userName: ‘alice123’,
_id: ObjectId("58a4eb4a30c75625e00d2820")
},
fullDocument: {_id: ObjectId("58a4eb4a30c75625e00d2820"),
userName: ‘alice123’,
name: ‘Alice’
}
}
INSERT
{_id: (resumeToken),
operationType: ‘delete’,
ns: {db: ‘test’, coll: ‘foo’},
documentKey: {userName: ‘alice123’,
_id: ObjectId("58a4eb4a30c75625e00d2820")
}
}
DELETE
{_id: (resumeToken),
operationType: ‘replace’,
ns: {db: ‘test’, coll: ‘foo’},
documentKey: {userName: ‘alice123’,
_id: ObjectId("58a4eb4a30c75625e00d2820")
},
fullDocument: {_id: ObjectId("58a4eb4a30c75625e00d2820"),
userName: ‘alice123’,
name: ‘Alice’
}
}
REPLACE
{_id: (resumeToken),
operationType: ‘update’,
ns: {db: ‘test’, coll: ‘foo’},
documentKey: {userName: ‘alice123’,
_id: ObjectId("58a4eb4a30c75625e00d2820")
},
updateDescription: {updatedFields: {email: ‘alice@10gen.com’},
removedFields: [‘phoneNumber’]}
}
UPDATE
{_id: (resumeToken),
operationType: ‘invalidate’,
ns: {db: ‘test’, coll: ‘foo’}}
INVALIDATE](https://image.slidesharecdn.com/1200-1240-using-change-strea-c3bf1aea-cfd8-4491-870e-0aab2523c7fa-2139307395-171018133356/85/Using-Change-Streams-to-Keep-Up-with-Your-Data-16-320.jpg)
![# M D B l o c a l
{_id: (resumeToken),
operationType: ‘update’,
ns: {db: ‘test’, coll: ‘foo’},
documentKey: {userName: ‘alice123’,
_id: ObjectId("58a4eb4a30c75625e00d2820")},
updateDescription: {updatedFields: {email: ‘alice@10gen.com’},
removedFields: [‘phoneNumber’]}
}
UPDATE
var changeStream = coll.watch({ fullDocument: 'updateLookup' });
UPDATE WITH FullDocument:
updateLookup
{_id: (resumeToken),
operationType: ‘update’,
ns: {db: ‘test’, coll: ‘foo’},
documentKey: {userName: ‘alice123’,
_id: ObjectId("58a4eb4a30c75625e00d2820")},
updateDescription: {updatedFields: {email: ‘alice@10gen.com’},
removedFields: [‘phoneNumber’]},
fullDocument: { _id: ObjectId("58a4eb4a30c75625e00d2820"),
name: ‘Alice’,
userName: ‘alice123’,
email: ‘alice@10gen.com’,
team: ‘replication’}
}](https://image.slidesharecdn.com/1200-1240-using-change-strea-c3bf1aea-cfd8-4491-870e-0aab2523c7fa-2139307395-171018133356/85/Using-Change-Streams-to-Keep-Up-with-Your-Data-17-320.jpg)
![# M D B l o c a l
Action Handler
{_id: (resumeToken),
operationType: ‘update’,
ns: {db: ‘data’, coll: ‘temperature’},
documentKey: {_id: ‘greenRoom123’)},
updateDescription: {updatedFields: {temperature: 68}
fullDocument: {_id: ‘greenRoom123’,
temperature: 68,
username: aly_cabral}
}
{_id: (resumeToken),
operationType: ‘insert’,
ns: {db: ‘data’, coll: ‘temperature’},
documentKey: {_id: ‘orangeRoom123’
fullDocument: {_id: ‘orangeRoom123’,
temperature: 65,
username: eliot_horowitz }
}
{_id: (resumeToken),
operationType: ‘insert’,
ns: {db: ‘data’, coll: ‘temperature’},
documentKey: {_id: ‘greenRoom123’)},
fullDocument: {_id: ‘greenRoom123’),
temperature: 75,
username: aly_cabral}
}
var changeStream = coll.watch([{
$match: {{ 'fullDocument.temperature': {$gte : 70 }}],
{fullDocument: 'updateLookup' });
temperature: 75
username: aly_cabral](https://image.slidesharecdn.com/1200-1240-using-change-strea-c3bf1aea-cfd8-4491-870e-0aab2523c7fa-2139307395-171018133356/85/Using-Change-Streams-to-Keep-Up-with-Your-Data-21-320.jpg)
![# M D B l o c a l
Action Handler
{_id: (resumeToken),
operationType: ‘replace’,
ns: {db: ‘data’, coll: ‘temperature’},
documentKey: {_id: ‘blueRoom123’},
fullDocument: {_id: ‘blueRoom123’,
temperature: 71,
username: asya_kamsky}
}
var changeStream = coll.watch([{ $match: {{ 'fullDocument.username': ‘aly_cabral’ }}],
{fullDocument: 'updateLookup' });
temperature: 75
username: aly_cabral
temperature: 85
username: aly_cabral
{_id: (resumeToken),
operationType: ‘delete’,
ns: {db: ‘data’, coll: ‘temperature’},
documentKey: {_id: ‘blueRoom123’}}
{_id: (resumeToken),
operationType: ‘insert’,
ns: {db: ‘data’, coll: ‘temperature’},
documentKey: {_id: ‘blueRoom123’},
fullDocument: {_id: ‘blueRoom123’,
temperature: 85,
username: aly_cabral}
}](https://image.slidesharecdn.com/1200-1240-using-change-strea-c3bf1aea-cfd8-4491-870e-0aab2523c7fa-2139307395-171018133356/85/Using-Change-Streams-to-Keep-Up-with-Your-Data-23-320.jpg)
![# M D B l o c a l
var changeStream = coll.watch([{ $match: {
{ $or: [ { 'fullDocument.username': ’aly_cabral' },
{ operationType: 'delete' },
{ operationType: 'replace' }]}],
{fullDocument: 'updateLookup' });
Insert
Update
Delete
[ insert ]: [ { _id:5, username: ’aly_cabral', temperature: 75 } ]
[ delete ]: []](https://image.slidesharecdn.com/1200-1240-using-change-strea-c3bf1aea-cfd8-4491-870e-0aab2523c7fa-2139307395-171018133356/85/Using-Change-Streams-to-Keep-Up-with-Your-Data-25-320.jpg)
![# M D B l o c a l
var changeStream = coll.watch([{ $match: {
{ $or: [ { 'fullDocument.username': ‘aly_cabral' },
{ operationType: 'delete' },
{ operationType: 'replace' }]}],
{fullDocument: 'updateLookup' });
Insert Update Delete
In the update notification fullDocument: null
updateLookup
{_id: (resumeToken),
operationType: ‘update’,
ns: {db: ‘data’, coll: ‘temperature’},
documentKey: {userName: ‘greenRoom123’},
updateDescription: {updatedFields: {temperature: 78}},
fullDocument: {}](https://image.slidesharecdn.com/1200-1240-using-change-strea-c3bf1aea-cfd8-4491-870e-0aab2523c7fa-2139307395-171018133356/85/Using-Change-Streams-to-Keep-Up-with-Your-Data-26-320.jpg)
![# M D B l o c a l
Action Handler
var changeStream = coll.watch([{ $match: {{ ‘documentKey.username’: ‘aly_cabral’ }}],
{fullDocument: 'updateLookup' });
temperature: 75
username: aly_cabral
temperature: 85
username: aly_cabral
{_id: (resumeToken),
operationType: ‘delete’,
ns: {db: ‘data’, coll: ‘temperature’},
documentKey: {_id: ‘blueRoom123’,
username: “aly_cabral”)}}
{_id: (resumeToken),
operationType: ‘replace’,
ns: {db: ‘data’, coll: ‘temperature’},
documentKey: {_id: ‘greenRoom123’,
username: ‘aly_cabral’},
fullDocument: {_id: ‘greenRoom123’,
temperature: 71,
username: “asya_kamsky”}
}
{_id: (resumeToken),
operationType: ‘insert’,
ns: {db: ‘data’, coll: ‘temperature’},
documentKey: {_id: ‘blueRoom123’,
username: “aly_cabral”},
fullDocument: {_id: ‘blueRoom123’,
temperature: 85,
username: “aly_cabral”}
}](https://image.slidesharecdn.com/1200-1240-using-change-strea-c3bf1aea-cfd8-4491-870e-0aab2523c7fa-2139307395-171018133356/85/Using-Change-Streams-to-Keep-Up-with-Your-Data-28-320.jpg)
Using Change Streams to Keep Up with Your Data
- 1. O C T O B E R 1 2 , 2 0 1 7 | B E S P O K E | S A N F R A N C I S C O # M D B l o c a l USING CHANGE STREAMS TO KEEP UP WITH YOUR DATA
- 2. # M D B l o c a l ALYSON CABRAL P R O D U C T M AN A G E R | M O N G O D B | @ ALY _ C A B R A L
- 3. # M D B l o c a l “TIME IS MONEY.” B E N J A M I N F R A N K L I NWA R R E N B U F F E T
- 4. # M D B l o c a l REAL-TIME IS AN EXPECTATION
- 5. # M D B l o c a l MongoDB Action Handler
- 6. # M D B l o c a l const changeStream= db.collection(‘temperature’).watch(); c h a n g e S t r e a m . o n ( ' c h a n g e ' , f u n c t i o n ( c h a n g e ) { c o n s o l e . l o g ( c h a n g e ) ; } ) ;
- 7. # M D B l o c a l 5 OPERATION TYPES INSERT DELETE REPLACE UPDATE INVALIDATE
- 8. # M D B l o c a l CHANGE STREAM CHARACTERISTICS
- 9. # M D B l o c a l CHANGE STREAMS UTILIZE COLLECTION ACCESS CONTROLS, PRESENT A DEFINED API, AND ENABLE SCALING ACROSS PRIMARIES AND SECONDARIES.
- 10. # M D B l o c a l PS CHANGES ARE DURABLE P S
- 11. # M D B l o c a l SP CHANGE STREAMS ARE RESUMABLE P S {_id: <resumeToken>, operationType: ‘update’, ns: {db: ‘data’, coll: ‘temperature’}, documentKey: {_id: ‘room12345’}, updateDescription: {updatedFields: {temperature: 70}, removedFields: [‘humidity’]} } var newChangeStream = coll.watch({ resumeAfter: <cachedResumeToken> });
- 12. # M D B l o c a l TOTAL ORDERING OF CHANGES ACROSS SHARDS mongos 3 1 2 Shard 1 Shard 2 Shard 3
- 13. # M D B l o c a l CHANGE STREAMS UTILIZE THE POWER OF THE AGGREGATION FRAMEWORK $match $project $addFields $replaceRoot $redact var changeStream = coll.watch([{ $match: { $or: [ { operationType: 'delete' }, { operationType: 'replace' } ] });
- 14. # M D B l o c a l 1. COLLECTION ACCESS CONTROLS 2. DEFINED API 3. ENABLE SCALING 4. DURABLE 5. RESUMABLE 6. TOTAL ORDERING 7. POWER OF AGGREGATION
- 15. # M D B l o c a l OPERATION TYPES
- 16. # M D B l o c a l 5 OPERATION TYPES { _id: (resumeToken), operationType: ‘insert’, ns: {db: ‘test’, coll: ‘foo’}, documentKey: {userName: ‘alice123’, _id: ObjectId("58a4eb4a30c75625e00d2820") }, fullDocument: {_id: ObjectId("58a4eb4a30c75625e00d2820"), userName: ‘alice123’, name: ‘Alice’ } } INSERT {_id: (resumeToken), operationType: ‘delete’, ns: {db: ‘test’, coll: ‘foo’}, documentKey: {userName: ‘alice123’, _id: ObjectId("58a4eb4a30c75625e00d2820") } } DELETE {_id: (resumeToken), operationType: ‘replace’, ns: {db: ‘test’, coll: ‘foo’}, documentKey: {userName: ‘alice123’, _id: ObjectId("58a4eb4a30c75625e00d2820") }, fullDocument: {_id: ObjectId("58a4eb4a30c75625e00d2820"), userName: ‘alice123’, name: ‘Alice’ } } REPLACE {_id: (resumeToken), operationType: ‘update’, ns: {db: ‘test’, coll: ‘foo’}, documentKey: {userName: ‘alice123’, _id: ObjectId("58a4eb4a30c75625e00d2820") }, updateDescription: {updatedFields: {email: ‘[email protected]’}, removedFields: [‘phoneNumber’]} } UPDATE {_id: (resumeToken), operationType: ‘invalidate’, ns: {db: ‘test’, coll: ‘foo’}} INVALIDATE
- 17. # M D B l o c a l {_id: (resumeToken), operationType: ‘update’, ns: {db: ‘test’, coll: ‘foo’}, documentKey: {userName: ‘alice123’, _id: ObjectId("58a4eb4a30c75625e00d2820")}, updateDescription: {updatedFields: {email: ‘[email protected]’}, removedFields: [‘phoneNumber’]} } UPDATE var changeStream = coll.watch({ fullDocument: 'updateLookup' }); UPDATE WITH FullDocument: updateLookup {_id: (resumeToken), operationType: ‘update’, ns: {db: ‘test’, coll: ‘foo’}, documentKey: {userName: ‘alice123’, _id: ObjectId("58a4eb4a30c75625e00d2820")}, updateDescription: {updatedFields: {email: ‘[email protected]’}, removedFields: [‘phoneNumber’]}, fullDocument: { _id: ObjectId("58a4eb4a30c75625e00d2820"), name: ‘Alice’, userName: ‘alice123’, email: ‘[email protected]’, team: ‘replication’} }
- 18. # M D B l o c a l mongos 3 1 2 Shard 1 Shard 2 Shard 3 1 . S H AR D K E Y I S U S E D TO R O U T E O P E R AT I O N S 2 . _ I D U N I Q U E N E S S I S E N F O R C E D B Y S H AR D 3 . A U N I Q U E I D E N T I F I E R F O R T H E D O C U M E N T I N A S H AR D E D C O L L E C T I O N I S T H E C O M B I N AT I O N O F _ I D AN D S H AR D K E Y
- 19. # M D B l o c a l BACK TO THE USE CASE
- 20. # M D B l o c a l
- 21. # M D B l o c a l Action Handler {_id: (resumeToken), operationType: ‘update’, ns: {db: ‘data’, coll: ‘temperature’}, documentKey: {_id: ‘greenRoom123’)}, updateDescription: {updatedFields: {temperature: 68} fullDocument: {_id: ‘greenRoom123’, temperature: 68, username: aly_cabral} } {_id: (resumeToken), operationType: ‘insert’, ns: {db: ‘data’, coll: ‘temperature’}, documentKey: {_id: ‘orangeRoom123’ fullDocument: {_id: ‘orangeRoom123’, temperature: 65, username: eliot_horowitz } } {_id: (resumeToken), operationType: ‘insert’, ns: {db: ‘data’, coll: ‘temperature’}, documentKey: {_id: ‘greenRoom123’)}, fullDocument: {_id: ‘greenRoom123’), temperature: 75, username: aly_cabral} } var changeStream = coll.watch([{ $match: {{ 'fullDocument.temperature': {$gte : 70 }}], {fullDocument: 'updateLookup' }); temperature: 75 username: aly_cabral
- 22. # M D B l o c a l ALWAYS MATCH ON UNCHANGING FIELDS
- 23. # M D B l o c a l Action Handler {_id: (resumeToken), operationType: ‘replace’, ns: {db: ‘data’, coll: ‘temperature’}, documentKey: {_id: ‘blueRoom123’}, fullDocument: {_id: ‘blueRoom123’, temperature: 71, username: asya_kamsky} } var changeStream = coll.watch([{ $match: {{ 'fullDocument.username': ‘aly_cabral’ }}], {fullDocument: 'updateLookup' }); temperature: 75 username: aly_cabral temperature: 85 username: aly_cabral {_id: (resumeToken), operationType: ‘delete’, ns: {db: ‘data’, coll: ‘temperature’}, documentKey: {_id: ‘blueRoom123’}} {_id: (resumeToken), operationType: ‘insert’, ns: {db: ‘data’, coll: ‘temperature’}, documentKey: {_id: ‘blueRoom123’}, fullDocument: {_id: ‘blueRoom123’, temperature: 85, username: aly_cabral} }
- 24. # M D B l o c a l ALWAYS HANDLE DELETES AND REPLACES APPROPRIATELY
- 25. # M D B l o c a l var changeStream = coll.watch([{ $match: { { $or: [ { 'fullDocument.username': ’aly_cabral' }, { operationType: 'delete' }, { operationType: 'replace' }]}], {fullDocument: 'updateLookup' }); Insert Update Delete [ insert ]: [ { _id:5, username: ’aly_cabral', temperature: 75 } ] [ delete ]: []
- 26. # M D B l o c a l var changeStream = coll.watch([{ $match: { { $or: [ { 'fullDocument.username': ‘aly_cabral' }, { operationType: 'delete' }, { operationType: 'replace' }]}], {fullDocument: 'updateLookup' }); Insert Update Delete In the update notification fullDocument: null updateLookup {_id: (resumeToken), operationType: ‘update’, ns: {db: ‘data’, coll: ‘temperature’}, documentKey: {userName: ‘greenRoom123’}, updateDescription: {updatedFields: {temperature: 78}}, fullDocument: {}
- 27. # M D B l o c a l HAVE MATCHING FIELDS IN THE DOCUMENT KEY
- 28. # M D B l o c a l Action Handler var changeStream = coll.watch([{ $match: {{ ‘documentKey.username’: ‘aly_cabral’ }}], {fullDocument: 'updateLookup' }); temperature: 75 username: aly_cabral temperature: 85 username: aly_cabral {_id: (resumeToken), operationType: ‘delete’, ns: {db: ‘data’, coll: ‘temperature’}, documentKey: {_id: ‘blueRoom123’, username: “aly_cabral”)}} {_id: (resumeToken), operationType: ‘replace’, ns: {db: ‘data’, coll: ‘temperature’}, documentKey: {_id: ‘greenRoom123’, username: ‘aly_cabral’}, fullDocument: {_id: ‘greenRoom123’, temperature: 71, username: “asya_kamsky”} } {_id: (resumeToken), operationType: ‘insert’, ns: {db: ‘data’, coll: ‘temperature’}, documentKey: {_id: ‘blueRoom123’, username: “aly_cabral”}, fullDocument: {_id: ‘blueRoom123’, temperature: 85, username: “aly_cabral”} }
- 29. # M D B l o c a l UNDERLYING TECHNOLOGY
- 30. # M D B l o c a l P S S S S
- 31. # M D B l o c a l 1234
- 32. # M D B l o c a l 190 203 207 233 278 304 C AC H E D R E S U M E TO K E N : 1 9 0
- 33. # M D B l o c a l YOU MADE IT!
Editor's Notes
- #2: Hey guys, thank you all for being here. I know it can get a little bit rough at this time at a full day event, so I appreciate you guys sticking with me. I promise you won’t regret it. Today, I have the privilege of introducing Change Streams, a new feature in MongoDB 3.6 (being released later this year)
- #4: Before we get into the fun stuff, I have a little story. I was at a trivia night recently and one of the questions was who popularized the term “time is money”. My guess, naturally, was Warren Buffet as the was the famous business man the popped into my head at the time. Well, turns out I was only a couple centuries off. The real answer is Benjamin Franklin. Benjamin Franklin popularized the term time is money in the1700s. What a fascinating phrase it feels weirdly modern doesn’t it? I think it’s because this phrase has never rang more true than it is today. With the evolution of technology, and enabled by these innovations, each of us is able to do more and more in every second. We live in an era where we can catch up on the lastest news, socialize with our friends, get a date for the evening -- all in the two minutes it takes for us to use the restroom. While that is a silly example, we have become accustomed to doing and reacting to things as quickly as possible.
- #5: As the builders we are in this room, we’ve realized that real-time is no longer this flashy nice-to-have in our applications. It is our users expectation. Whether those users are our bosses looking at real-time monitoring dashboard or the general public wanting updates on their food deliveries precise location. We at MongoDB sought out to make it easier for you to rise to the high expectations your users have. And with that, in MongoDB 3.6 we are introducing Change Streams. In this talk, I will give you a simple definition of what change streams are and the power they possess. Then we will go into some one the charactertics change streams have. Finally, we will discuss how you can avoid some potential pitfalls.
- #6: So let’s start with a simple definition. Change streams allow you to watch all the changes against a given collection. So, when you open change streams against a collection you will see all of the changes applied to documents that belong to that collection. On each change a listening application will be notified with a document describing the change. Imagine this scenario, I have a smart thermometer within my apartment. Every second it inserts a document into MongoDB with temperature data. Change streams can then notify any listening application to every change in my temperature collection. So as the thermometer inserts new data, change streams will inform my listening action handler. In this case, the action handler is an application I built to turn on a fan every time the temperature goes above 70 degrees. Don’t let the simplicity of this example fool you, this concept can be applied to back-office management applications to end user applications. Change streams are extremely powerful
- #7: So, this slide is to show you how incredibly simple it is to implement change streams in your application. This example happens to be in node. Here, I am defining a change stream. Then starting it where every change gets printed out. It’s extremely simple and incredibly powerful. With change stream, you get a real-time feed of the changes to your data, enabling your application to react to these changes immediately.
- #8: Now we are going to move into some of the characteristics of change streams.
- #17: Now we are going to move into some of the characteristics of change streams.
- #31: The oplog is an internal mechanism that tracks all changes across the entire cluster, whether they are data or system changes. The