SlideShare a Scribd company logo
MongoDB .local Munich 2019: Best Practices for Working with IoT and Time-series Data
@
#MDBlocal
Best practices:
Working with IoT and Time-Series
Data
Robert Walters, Director of IoT, MongoDB
@RobsCranium
MUNICH
LinkedIn: https://www.linkedin.com/in/robwaltersprofile/
#MDBLocal
#MDBLocal
Agenda
Define time-series data
Review typical IoT Architecture
Designing time-series schemas
Query and visualize time-series data
#MDBLocal
What is time series data?
Data in an IoT Architecture
#MDBLocal
https://tinyurl.com/Mongodb-Houston-2019
https://www.mongodb.com/white-papers
#MDBLocal
Common IoT Architecture
#MDBLocal
Common IoT Architecture
#MDBLocal
Common IoT Architecture
#MDBLocal
Common IoT Architecture
#MDBLocal
Common IoT Architecture
#MDBLocal
Common IoT Architecture
#MDBLocal
MongoDB IoT Reference Architecture
#MDBLocal
Data at the edge
Designing time-series schemas
#MDBLocal
Breaking the relational mindset….
#MDBLocal
MongoDB Schema Design
for Time Series data
#MDBLocal
Effects on Document count
Per second vs Per Minute
#MDBLocal
Effects on data storage
Per second vs Per Minute
#MDBLocal
Effects on memory
Per second vs Per Minute
#MDBLocal
Sample code to add to the size based bucketMongoDB Schema Design
for Time Series data
#MDBLocal
Case Study:
https://www.ampcontrolgroup.com/
Leaders in industrial electrical and electronic
engineering
Selected MongoDB for Gas-Detection Mining Application
#MDBLocal
Case Study:
#MDBLocal
Case Study:
One Document per Device per Parameter per Hour
Pre-aggregated data updated for every update
After 3 months samples array removed
Data archived for 7 years
#MDBLocal
Keys to successful time series applications
Know your application requirements
Writes Reads Data Retention Security
#MDBLocal
What to do with old data
• Pre-aggregation
• Offline archival
• db.remove
• TTL Indexes
• Dropping the collection
• Online archival
• Queryable backups
• Sharding archival data
• Offload to data warehouse / S3
#MDBLocal
MongoDB Atlas Data Lake
Analyze data in any format on S3 using MongoDB Query Language
Archive IoT and Time-series data for long term storage and analysis
How to query and visualize
time-series data
#MDBLocal
Real-time Analytics
One database for operational and analytical
data
Easy to query unstructured data
MongoDB Query Language (MQL)
Aggregation Framework
#MDBLocal
Real-time Analytics
Easily query unstructured data with Aggregation Framework
#MDBLocal
Using MQL Demo
What is the largest value for a given sensor id?
db.SensorData.aggregate( [
{$match: { "sensorid":3003 }},
{$unwind: { path: '$samples' }},
{$group: { _id: '$day', MaxValue: { $max:
'$samples.val' } }},
{$sort: { "id": 1 }} ])
What is the largest value for a given sensor id?
db.SensorData.aggregate( [
{$match: { "sensorid":3003 }},
{$unwind: { path: '$samples' }},
{$group: { _id: '$day', MaxValue: { $max:
'$samples.val' } }},
{$sort: { "id": 1 }} ])
What is the largest value for a given sensor id?
db.SensorData.aggregate( [
{$match: { "sensorid":3003 }},
{$unwind: { path: '$samples' }},
{$group: { _id: '$day', MaxValue: { $max:
'$samples.val' } }},
{$sort: { "id": 1 }} ])
What is the largest value for a given sensor id?
db.SensorData.aggregate( [
{$match: { "sensorid":3003 }},
{$unwind: { path: '$samples' }},
{$group: { _id: '$day', MaxValue: { $max:
'$samples.val' } }},
{$sort: { "id": 1 }} ])
Cloud – Real-time Analytics
What is the largest value for a given sensor id?
db.SensorData.aggregate( [
{$match: { "sensorid":3003 }},
{$unwind: { path: '$samples' }},
{$group: { id: '$day', MaxValue: { $max:
'$samples.val' } }},
{$sort: { "id": 1 }} ])
{ "_id" : ISODate("2018-12-10T00:00:00Z"), "MaxValue" : 40.4 }
{ "_id" : ISODate("2018-12-11T00:00:00Z"), "MaxValue" : 40.43 }
{ "_id" : ISODate("2018-12-12T00:00:00Z"), "MaxValue" : 40.09 }
{ "_id" : ISODate("2018-12-13T00:00:00Z"), "MaxValue" : 39.7 }
…
#MDBLocal
Cloud – Visualization
MongoDB Charts
#MDBLocal
Cloud – Visualization
MongoDB BI Connector
#MDBLocal
Integration with
Why MongoDB for IoT and Time-
Series Data?
#MDBLocal
Why MongoDB for IoT Data?
#MDBLocal
Best practices summary
Know your requirements
1 sensor reading per document works but not the most efficient
Data retention policy
Data access requirements
For very large data consider multiple collections
Faster to archive/drop collections than db.remove()
Recommended design fixed-based schema
https://www.mongodb.com/collateral/time-series-best-practices
Archive data to S3 / Atlas Data Lake for long term storage and analysis
#MDBLocal
Resources
MongoDB IoT Reference Architecture
Time Series Data and MongoDB: Best Practices Guide
Get Started with your project in MongoDB Atlas.
Use code “IOT200” for $200 in credit!
Thinking about an IoT Solution?
Robert.Walters@MongoDB.COM
THANK YOU
MongoDB .local Munich 2019: Best Practices for Working with IoT and Time-series Data

More Related Content

What's hot (20)

PDF
Webinar: Building Your First App with MongoDB and Java
MongoDB
 
PDF
MongoDB .local Toronto 2019: Tips and Tricks for Effective Indexing
MongoDB
 
PPTX
Joins and Other Aggregation Enhancements Coming in MongoDB 3.2
MongoDB
 
PPTX
Back to Basics Webinar 3: Schema Design Thinking in Documents
MongoDB
 
PPTX
Socialite, the Open Source Status Feed Part 2: Managing the Social Graph
MongoDB
 
PPTX
Back to Basics Webinar 4: Advanced Indexing, Text and Geospatial Indexes
MongoDB
 
PPTX
Webinar: Schema Design
MongoDB
 
PDF
MongoDB .local Chicago 2019: Best Practices for Working with IoT and Time-ser...
MongoDB
 
PPTX
High Performance Applications with MongoDB
MongoDB
 
PDF
MongoDB Schema Design
Alex Litvinok
 
KEY
Managing Social Content with MongoDB
MongoDB
 
PPTX
Back to Basics Webinar 2: Your First MongoDB Application
MongoDB
 
PPTX
Webinar: Transitioning from SQL to MongoDB
MongoDB
 
PPTX
2014 bigdatacamp asya_kamsky
Data Con LA
 
PDF
MongoDB Launchpad 2016: What’s New in the 3.4 Server
MongoDB
 
KEY
Schema design
christkv
 
PPTX
Indexing Strategies to Help You Scale
MongoDB
 
PPTX
User Data Management with MongoDB
MongoDB
 
PPTX
Building a Scalable Inbox System with MongoDB and Java
antoinegirbal
 
PPTX
Building a Location-based platform with MongoDB from Zero.
Ravi Teja
 
Webinar: Building Your First App with MongoDB and Java
MongoDB
 
MongoDB .local Toronto 2019: Tips and Tricks for Effective Indexing
MongoDB
 
Joins and Other Aggregation Enhancements Coming in MongoDB 3.2
MongoDB
 
Back to Basics Webinar 3: Schema Design Thinking in Documents
MongoDB
 
Socialite, the Open Source Status Feed Part 2: Managing the Social Graph
MongoDB
 
Back to Basics Webinar 4: Advanced Indexing, Text and Geospatial Indexes
MongoDB
 
Webinar: Schema Design
MongoDB
 
MongoDB .local Chicago 2019: Best Practices for Working with IoT and Time-ser...
MongoDB
 
High Performance Applications with MongoDB
MongoDB
 
MongoDB Schema Design
Alex Litvinok
 
Managing Social Content with MongoDB
MongoDB
 
Back to Basics Webinar 2: Your First MongoDB Application
MongoDB
 
Webinar: Transitioning from SQL to MongoDB
MongoDB
 
2014 bigdatacamp asya_kamsky
Data Con LA
 
MongoDB Launchpad 2016: What’s New in the 3.4 Server
MongoDB
 
Schema design
christkv
 
Indexing Strategies to Help You Scale
MongoDB
 
User Data Management with MongoDB
MongoDB
 
Building a Scalable Inbox System with MongoDB and Java
antoinegirbal
 
Building a Location-based platform with MongoDB from Zero.
Ravi Teja
 

Similar to MongoDB .local Munich 2019: Best Practices for Working with IoT and Time-series Data (20)

PDF
MongoDB .local London 2019: Best Practices for Working with IoT and Time-seri...
MongoDB
 
PDF
MongoDB .local Houston 2019: Best Practices for Working with IoT and Time-ser...
MongoDB
 
PDF
MongoDB .local Paris 2020: Les bonnes pratiques pour travailler avec les donn...
MongoDB
 
PDF
MongoDB.local DC 2018: Tutorial - Data Analytics with MongoDB
MongoDB
 
PPTX
How to leverage what's new in MongoDB 3.6
Maxime Beugnet
 
PDF
Simplifying & accelerating application development with MongoDB's intelligent...
Maxime Beugnet
 
PPTX
Eagle6 mongo dc revised
MongoDB
 
PPTX
Eagle6 Enterprise Situational Awareness
MongoDB
 
PDF
Analytics with MongoDB Aggregation Framework and Hadoop Connector
Henrik Ingo
 
PPTX
Marc s01 e02-crud-database
MongoDB
 
PPTX
Webinarserie: Einführung in MongoDB: “Back to Basics” - Teil 3 - Interaktion ...
MongoDB
 
PPTX
[MongoDB.local Bengaluru 2018] Introduction to MongoDB Stitch
MongoDB
 
PPTX
Sharing about MongoDB Overview and Indexing in MongoDB
TrnLTunKit
 
PDF
MongoDB Meetup
Maxime Beugnet
 
PPTX
S01 e01 schema-design
MongoDB
 
PPTX
Socialite, the Open Source Status Feed
MongoDB
 
PDF
Montreal Elasticsearch Meetup
Loïc Bertron
 
PPTX
Using MongoDB As a Tick Database
MongoDB
 
PPTX
[MongoDB.local Bengaluru 2018] Keynote
MongoDB
 
PDF
Lab pratico per la progettazione di soluzioni MongoDB in ambito Internet of T...
festival ICT 2016
 
MongoDB .local London 2019: Best Practices for Working with IoT and Time-seri...
MongoDB
 
MongoDB .local Houston 2019: Best Practices for Working with IoT and Time-ser...
MongoDB
 
MongoDB .local Paris 2020: Les bonnes pratiques pour travailler avec les donn...
MongoDB
 
MongoDB.local DC 2018: Tutorial - Data Analytics with MongoDB
MongoDB
 
How to leverage what's new in MongoDB 3.6
Maxime Beugnet
 
Simplifying & accelerating application development with MongoDB's intelligent...
Maxime Beugnet
 
Eagle6 mongo dc revised
MongoDB
 
Eagle6 Enterprise Situational Awareness
MongoDB
 
Analytics with MongoDB Aggregation Framework and Hadoop Connector
Henrik Ingo
 
Marc s01 e02-crud-database
MongoDB
 
Webinarserie: Einführung in MongoDB: “Back to Basics” - Teil 3 - Interaktion ...
MongoDB
 
[MongoDB.local Bengaluru 2018] Introduction to MongoDB Stitch
MongoDB
 
Sharing about MongoDB Overview and Indexing in MongoDB
TrnLTunKit
 
MongoDB Meetup
Maxime Beugnet
 
S01 e01 schema-design
MongoDB
 
Socialite, the Open Source Status Feed
MongoDB
 
Montreal Elasticsearch Meetup
Loïc Bertron
 
Using MongoDB As a Tick Database
MongoDB
 
[MongoDB.local Bengaluru 2018] Keynote
MongoDB
 
Lab pratico per la progettazione di soluzioni MongoDB in ambito Internet of T...
festival ICT 2016
 
Ad

More from MongoDB (20)

PDF
MongoDB SoCal 2020: Migrate Anything* to MongoDB Atlas
MongoDB
 
PDF
MongoDB SoCal 2020: Go on a Data Safari with MongoDB Charts!
MongoDB
 
PDF
MongoDB SoCal 2020: Using MongoDB Services in Kubernetes: Any Platform, Devel...
MongoDB
 
PDF
MongoDB SoCal 2020: A Complete Methodology of Data Modeling for MongoDB
MongoDB
 
PDF
MongoDB SoCal 2020: From Pharmacist to Analyst: Leveraging MongoDB for Real-T...
MongoDB
 
PDF
MongoDB SoCal 2020: MongoDB Atlas Jump Start
MongoDB
 
PDF
MongoDB .local San Francisco 2020: Powering the new age data demands [Infosys]
MongoDB
 
PDF
MongoDB .local San Francisco 2020: Using MongoDB Services in Kubernetes: any ...
MongoDB
 
PDF
MongoDB .local San Francisco 2020: Go on a Data Safari with MongoDB Charts!
MongoDB
 
PDF
MongoDB .local San Francisco 2020: MongoDB Atlas Jumpstart
MongoDB
 
PDF
MongoDB .local San Francisco 2020: Tips and Tricks++ for Querying and Indexin...
MongoDB
 
PDF
MongoDB .local San Francisco 2020: Aggregation Pipeline Power++
MongoDB
 
PDF
MongoDB .local San Francisco 2020: A Complete Methodology of Data Modeling fo...
MongoDB
 
PDF
MongoDB .local San Francisco 2020: MongoDB Atlas Data Lake Technical Deep Dive
MongoDB
 
PDF
MongoDB .local San Francisco 2020: Developing Alexa Skills with MongoDB & Golang
MongoDB
 
PDF
MongoDB .local Paris 2020: Realm : l'ingrédient secret pour de meilleures app...
MongoDB
 
PDF
MongoDB .local Paris 2020: Upply @MongoDB : Upply : Quand le Machine Learning...
MongoDB
 
PDF
MongoDB .local Paris 2020: Les bonnes pratiques pour sécuriser MongoDB
MongoDB
 
PDF
MongoDB .local Paris 2020: Tout savoir sur le moteur de recherche Full Text S...
MongoDB
 
PDF
MongoDB .local Paris 2020: Adéo @MongoDB : MongoDB Atlas & Leroy Merlin : et ...
MongoDB
 
MongoDB SoCal 2020: Migrate Anything* to MongoDB Atlas
MongoDB
 
MongoDB SoCal 2020: Go on a Data Safari with MongoDB Charts!
MongoDB
 
MongoDB SoCal 2020: Using MongoDB Services in Kubernetes: Any Platform, Devel...
MongoDB
 
MongoDB SoCal 2020: A Complete Methodology of Data Modeling for MongoDB
MongoDB
 
MongoDB SoCal 2020: From Pharmacist to Analyst: Leveraging MongoDB for Real-T...
MongoDB
 
MongoDB SoCal 2020: MongoDB Atlas Jump Start
MongoDB
 
MongoDB .local San Francisco 2020: Powering the new age data demands [Infosys]
MongoDB
 
MongoDB .local San Francisco 2020: Using MongoDB Services in Kubernetes: any ...
MongoDB
 
MongoDB .local San Francisco 2020: Go on a Data Safari with MongoDB Charts!
MongoDB
 
MongoDB .local San Francisco 2020: MongoDB Atlas Jumpstart
MongoDB
 
MongoDB .local San Francisco 2020: Tips and Tricks++ for Querying and Indexin...
MongoDB
 
MongoDB .local San Francisco 2020: Aggregation Pipeline Power++
MongoDB
 
MongoDB .local San Francisco 2020: A Complete Methodology of Data Modeling fo...
MongoDB
 
MongoDB .local San Francisco 2020: MongoDB Atlas Data Lake Technical Deep Dive
MongoDB
 
MongoDB .local San Francisco 2020: Developing Alexa Skills with MongoDB & Golang
MongoDB
 
MongoDB .local Paris 2020: Realm : l'ingrédient secret pour de meilleures app...
MongoDB
 
MongoDB .local Paris 2020: Upply @MongoDB : Upply : Quand le Machine Learning...
MongoDB
 
MongoDB .local Paris 2020: Les bonnes pratiques pour sécuriser MongoDB
MongoDB
 
MongoDB .local Paris 2020: Tout savoir sur le moteur de recherche Full Text S...
MongoDB
 
MongoDB .local Paris 2020: Adéo @MongoDB : MongoDB Atlas & Leroy Merlin : et ...
MongoDB
 
Ad

Recently uploaded (20)

PPTX
MuleSoft MCP Support (Model Context Protocol) and Use Case Demo
shyamraj55
 
PPTX
Agentforce World Tour Toronto '25 - MCP with MuleSoft
Alexandra N. Martinez
 
PDF
Reverse Engineering of Security Products: Developing an Advanced Microsoft De...
nwbxhhcyjv
 
PPT
Ericsson LTE presentation SEMINAR 2010.ppt
npat3
 
PDF
LOOPS in C Programming Language - Technology
RishabhDwivedi43
 
PDF
Kit-Works Team Study_20250627_한달만에만든사내서비스키링(양다윗).pdf
Wonjun Hwang
 
PPTX
Future Tech Innovations 2025 – A TechLists Insight
TechLists
 
PDF
The 2025 InfraRed Report - Redpoint Ventures
Razin Mustafiz
 
DOCX
Python coding for beginners !! Start now!#
Rajni Bhardwaj Grover
 
PPTX
From Sci-Fi to Reality: Exploring AI Evolution
Svetlana Meissner
 
PDF
Book industry state of the nation 2025 - Tech Forum 2025
BookNet Canada
 
PDF
“Squinting Vision Pipelines: Detecting and Correcting Errors in Vision Models...
Edge AI and Vision Alliance
 
PDF
Newgen Beyond Frankenstein_Build vs Buy_Digital_version.pdf
darshakparmar
 
DOCX
Cryptography Quiz: test your knowledge of this important security concept.
Rajni Bhardwaj Grover
 
PPTX
Seamless Tech Experiences Showcasing Cross-Platform App Design.pptx
presentifyai
 
PDF
UiPath DevConnect 2025: Agentic Automation Community User Group Meeting
DianaGray10
 
PDF
NASA A Researcher’s Guide to International Space Station : Physical Sciences ...
Dr. PANKAJ DHUSSA
 
PDF
Transcript: Book industry state of the nation 2025 - Tech Forum 2025
BookNet Canada
 
PPTX
New ThousandEyes Product Innovations: Cisco Live June 2025
ThousandEyes
 
PDF
UPDF - AI PDF Editor & Converter Key Features
DealFuel
 
MuleSoft MCP Support (Model Context Protocol) and Use Case Demo
shyamraj55
 
Agentforce World Tour Toronto '25 - MCP with MuleSoft
Alexandra N. Martinez
 
Reverse Engineering of Security Products: Developing an Advanced Microsoft De...
nwbxhhcyjv
 
Ericsson LTE presentation SEMINAR 2010.ppt
npat3
 
LOOPS in C Programming Language - Technology
RishabhDwivedi43
 
Kit-Works Team Study_20250627_한달만에만든사내서비스키링(양다윗).pdf
Wonjun Hwang
 
Future Tech Innovations 2025 – A TechLists Insight
TechLists
 
The 2025 InfraRed Report - Redpoint Ventures
Razin Mustafiz
 
Python coding for beginners !! Start now!#
Rajni Bhardwaj Grover
 
From Sci-Fi to Reality: Exploring AI Evolution
Svetlana Meissner
 
Book industry state of the nation 2025 - Tech Forum 2025
BookNet Canada
 
“Squinting Vision Pipelines: Detecting and Correcting Errors in Vision Models...
Edge AI and Vision Alliance
 
Newgen Beyond Frankenstein_Build vs Buy_Digital_version.pdf
darshakparmar
 
Cryptography Quiz: test your knowledge of this important security concept.
Rajni Bhardwaj Grover
 
Seamless Tech Experiences Showcasing Cross-Platform App Design.pptx
presentifyai
 
UiPath DevConnect 2025: Agentic Automation Community User Group Meeting
DianaGray10
 
NASA A Researcher’s Guide to International Space Station : Physical Sciences ...
Dr. PANKAJ DHUSSA
 
Transcript: Book industry state of the nation 2025 - Tech Forum 2025
BookNet Canada
 
New ThousandEyes Product Innovations: Cisco Live June 2025
ThousandEyes
 
UPDF - AI PDF Editor & Converter Key Features
DealFuel
 

MongoDB .local Munich 2019: Best Practices for Working with IoT and Time-series Data