MongoDB .local Chicago 2019: Best Practices for Working with IoT and Time-series Data
1 like268 views
The document outlines best practices for working with IoT and time-series data, focusing on designing schemas, querying, and visualizing such data using MongoDB. It discusses the importance of understanding application requirements, data retention, and efficient data management strategies like pre-aggregation and archival. Additionally, it provides case studies and highlights MongoDB's capabilities for handling real-time analytics and unstructured data.
![What is the largest value for a given sensor id?
db.SensorData.aggregate( [
{$match: { "sensorid":3003 }},
{$unwind: { path: '$samples' }},
{$group: { _id: '$day', MaxValue: { $max:
'$samples.val' } }},
{$sort: { "id": 1 }} ])](https://image.slidesharecdn.com/dotlocalchicago-iottimeseries2019-191031155729/85/MongoDB-local-Chicago-2019-Best-Practices-for-Working-with-IoT-and-Time-series-Data-33-320.jpg)
![What is the largest value for a given sensor id?
db.SensorData.aggregate( [
{$match: { "sensorid":3003 }},
{$unwind: { path: '$samples' }},
{$group: { _id: '$day', MaxValue: { $max:
'$samples.val' } }},
{$sort: { "id": 1 }} ])](https://image.slidesharecdn.com/dotlocalchicago-iottimeseries2019-191031155729/85/MongoDB-local-Chicago-2019-Best-Practices-for-Working-with-IoT-and-Time-series-Data-34-320.jpg)
![What is the largest value for a given sensor id?
db.SensorData.aggregate( [
{$match: { "sensorid":3003 }},
{$unwind: { path: '$samples' }},
{$group: { _id: '$day', MaxValue: { $max:
'$samples.val' } }},
{$sort: { "id": 1 }} ])](https://image.slidesharecdn.com/dotlocalchicago-iottimeseries2019-191031155729/85/MongoDB-local-Chicago-2019-Best-Practices-for-Working-with-IoT-and-Time-series-Data-35-320.jpg)
![What is the largest value for a given sensor id?
db.SensorData.aggregate( [
{$match: { "sensorid":3003 }},
{$unwind: { path: '$samples' }},
{$group: { _id: '$day', MaxValue: { $max:
'$samples.val' } }},
{$sort: { "id": 1 }} ])](https://image.slidesharecdn.com/dotlocalchicago-iottimeseries2019-191031155729/85/MongoDB-local-Chicago-2019-Best-Practices-for-Working-with-IoT-and-Time-series-Data-36-320.jpg)
![Cloud – Real-time Analytics
What is the largest value for a given sensor id?
db.SensorData.aggregate( [
{$match: { "sensorid":3003 }},
{$unwind: { path: '$samples' }},
{$group: { id: '$day', MaxValue: { $max:
'$samples.val' } }},
{$sort: { "id": 1 }} ])
{ "_id" : ISODate("2018-12-10T00:00:00Z"), "MaxValue" : 40.4 }
{ "_id" : ISODate("2018-12-11T00:00:00Z"), "MaxValue" : 40.43 }
{ "_id" : ISODate("2018-12-12T00:00:00Z"), "MaxValue" : 40.09 }
{ "_id" : ISODate("2018-12-13T00:00:00Z"), "MaxValue" : 39.7 }
…](https://image.slidesharecdn.com/dotlocalchicago-iottimeseries2019-191031155729/85/MongoDB-local-Chicago-2019-Best-Practices-for-Working-with-IoT-and-Time-series-Data-37-320.jpg)
MongoDB .local Chicago 2019: Best Practices for Working with IoT and Time-series Data
- 2. @ #MDBlocal Best practices: Working with IoT and Time-Series Data Robert Walters, Director of IoT, MongoDB @RobsCranium CHICAGO LinkedIn: https://www.linkedin.com/in/robwaltersprofile/
- 3. #MDBLocal
- 4. #MDBLocal Agenda Define time-series data Review typical IoT Architecture Designing time-series schemas Query and visualize time-series data
- 5. #MDBLocal What is time series data?
- 6. Data in an IoT Architecture
- 14. #MDBLocal MongoDB IoT Reference Architecture
- 15. #MDBLocal Data at the edge
- 17. #MDBLocal Breaking the relational mindset….
- 18. #MDBLocal MongoDB Schema Design for Time Series data
- 19. #MDBLocal Effects on Document count Per second vs Per Minute
- 20. #MDBLocal Effects on data storage Per second vs Per Minute
- 21. #MDBLocal Effects on memory Per second vs Per Minute
- 22. #MDBLocal Sample code to add to the size based bucketMongoDB Schema Design for Time Series data
- 23. #MDBLocal Case Study: https://www.ampcontrolgroup.com/ Leaders in industrial electrical and electronic engineering Selected MongoDB for Gas-Detection Mining Application
- 25. #MDBLocal Case Study: One Document per Device per Parameter per Hour Pre-aggregated data updated for every update After 3 months samples array removed Data archived for 7 years
- 26. #MDBLocal Keys to successful time series applications Know your application requirements Writes Reads Data Retention Security
- 27. #MDBLocal What to do with old data • Pre-aggregation • Offline archival • db.remove • TTL Indexes • Dropping the collection • Online archival • Queryable backups • Sharding archival data • Offload to data warehouse / S3
- 28. #MDBLocal MongoDB Atlas Data Lake Analyze data in any format on S3 using MongoDB Query Language Archive IoT and Time-series data for long term storage and analysis
- 29. How to query and visualize time-series data
- 30. #MDBLocal Real-time Analytics One database for operational and analytical data Easy to query unstructured data MongoDB Query Language (MQL) Aggregation Framework
- 31. #MDBLocal Real-time Analytics Easily query unstructured data with Aggregation Framework
- 33. What is the largest value for a given sensor id? db.SensorData.aggregate( [ {$match: { "sensorid":3003 }}, {$unwind: { path: '$samples' }}, {$group: { _id: '$day', MaxValue: { $max: '$samples.val' } }}, {$sort: { "id": 1 }} ])
- 34. What is the largest value for a given sensor id? db.SensorData.aggregate( [ {$match: { "sensorid":3003 }}, {$unwind: { path: '$samples' }}, {$group: { _id: '$day', MaxValue: { $max: '$samples.val' } }}, {$sort: { "id": 1 }} ])
- 35. What is the largest value for a given sensor id? db.SensorData.aggregate( [ {$match: { "sensorid":3003 }}, {$unwind: { path: '$samples' }}, {$group: { _id: '$day', MaxValue: { $max: '$samples.val' } }}, {$sort: { "id": 1 }} ])
- 36. What is the largest value for a given sensor id? db.SensorData.aggregate( [ {$match: { "sensorid":3003 }}, {$unwind: { path: '$samples' }}, {$group: { _id: '$day', MaxValue: { $max: '$samples.val' } }}, {$sort: { "id": 1 }} ])
- 37. Cloud – Real-time Analytics What is the largest value for a given sensor id? db.SensorData.aggregate( [ {$match: { "sensorid":3003 }}, {$unwind: { path: '$samples' }}, {$group: { id: '$day', MaxValue: { $max: '$samples.val' } }}, {$sort: { "id": 1 }} ]) { "_id" : ISODate("2018-12-10T00:00:00Z"), "MaxValue" : 40.4 } { "_id" : ISODate("2018-12-11T00:00:00Z"), "MaxValue" : 40.43 } { "_id" : ISODate("2018-12-12T00:00:00Z"), "MaxValue" : 40.09 } { "_id" : ISODate("2018-12-13T00:00:00Z"), "MaxValue" : 39.7 } …
- 38. #MDBLocal Cloud – Visualization MongoDB Charts
- 39. #MDBLocal Cloud – Visualization MongoDB BI Connector
- 41. Why MongoDB for IoT and Time- Series Data?
- 42. #MDBLocal Why MongoDB for IoT Data?
- 43. #MDBLocal Best practices summary Know your requirements 1 sensor reading per document works but not the most efficient Data retention policy Data access requirements For very large data consider multiple collections Faster to archive/drop collections than db.remove() Recommended design fixed-based schema https://www.mongodb.com/collateral/time-series-best-practices Archive data to S3 / Atlas Data Lake for long term storage and analysis
- 44. #MDBLocal Resources MongoDB IoT Reference Architecture Time Series Data and MongoDB: Best Practices Guide Get Started with your project in MongoDB Atlas. Use code “IOT200” for $200 in credit! Thinking about an IoT Solution? [email protected]
- 46. THANK YOU