MongoDB .local London 2019: MongoDB Atlas Data Lake Technical Deep Dive
0 likes359 views
This document summarizes MongoDB Atlas Data Lake, a new service that allows customers to access, query, and analyze long-term data stored in AWS S3 buckets. It implements MongoDB's query language and security model to provide a familiar interface for working with structured data in object storage. The service is read-only, distributed, and optimized to handle queries over vast amounts of data efficiently using MongoDB's aggregation engine. Customers maintain full control over their data and how it is configured and accessed.
![history: {
customers: [{
store: "ent-archive",
definition: "/customers/*"
}],
invoices: [{
store: "ent-archive",
definition: "/invoices/{year int}/*"
}, {
store: "ent-archive",
definition: "/invoices/{year int}"
}]
}
Configuration: Data](https://image.slidesharecdn.com/mdbll19atlasdatalaketechnicaldeepdive-190925123419/85/MongoDB-local-London-2019-MongoDB-Atlas-Data-Lake-Technical-Deep-Dive-18-320.jpg)
![history: {
invoices: [{
store: "ent-archive",
definition: "/invoices/{year int}/*"
}, {
store: "ent-archive",
definition : "/invoices/{year int}”
}, {
store: "atlas",
cluster: "my-cluster",
db: "customers",
collection: "invoices"
}]
}
Configuration: Data (Future)](https://image.slidesharecdn.com/mdbll19atlasdatalaketechnicaldeepdive-190925123419/85/MongoDB-local-London-2019-MongoDB-Atlas-Data-Lake-Technical-Deep-Dive-19-320.jpg)
![{ $group: { _id: "$year", totalAvg: { $avg: "amount" } } }
Query Example: $group
Map:
{ $group: { _id: "$year",
totalAvg_sum: { $sum: "amount" },
totalAvg_count: { $sum: 1 }
} }
Reduce:
{ $group: { _id: "$_id",
totalAvg_sum: { $sum: "$totalAvg_sum" },
totalAvg_count: { $sum: "$totalAvg_count" }
} }
Finalize:
{ $project: { _id: "$_id", totalAvg: { $divide: ["$totalAvg_sum", "$totalAvg_count"] } } }](https://image.slidesharecdn.com/mdbll19atlasdatalaketechnicaldeepdive-190925123419/85/MongoDB-local-London-2019-MongoDB-Atlas-Data-Lake-Technical-Deep-Dive-24-320.jpg)
![#MDBlocal
MongoDB Atlas Data Lake
Technical Deep Dive [DEV]
James Osgood
https://www.surveymonkey.com/r/KQZZB8N](https://image.slidesharecdn.com/mdbll19atlasdatalaketechnicaldeepdive-190925123419/85/MongoDB-local-London-2019-MongoDB-Atlas-Data-Lake-Technical-Deep-Dive-33-320.jpg)
MongoDB .local London 2019: MongoDB Atlas Data Lake Technical Deep Dive
- 1. WHEN WHERE Atlas Data Lake Technical Deep-Dive James Osgood – Senior Solutions Architect Paul Done – Master Solutions Architect
- 2. #MDBLocal State of Affairs Why are we building this? • Businesses have a humongous amount of data • IDC predicts that by 2025 global data will reach 175 Zettabytes and 49% of it will reside in the public cloud. • Cloud storage is cost-effective • Cloud storage is hard to operationalize
- 3. #MDBLocal A New Service Offered by MongoDB Atlas Atlas Data Lake allows you to... ▪ Access long-term data ▪ Query long-term data ▪ Analyze long-term data
- 4. #MDBLocal Requirements Every product has requirements! ▪ Look and act like MongoDB ▪ Access customer’s data securely ▪ Handle queries over vast amounts of data ▪ Handle long-running queries ▪ Efficient use of resources
- 5. #MDBLocal Structured documents in AWS S3 Buckets Opens Up A Whole Ecosystem Of Data Science Tools By implementing MQL’s wire protocol, a rich & mature toolkit is available for data analytics MongoDB BI Connector MongoDB Compass MongoDB Charts3rd Party BI & Reporting Tools Including Machine Learning MongoDB’s standard programming language Drivers for use in custom scripts & applications for advanced analytics MongoDB’s Query Engine
- 7. #MDBLocal Language Must be able to communicate with our drivers. ▪ Written in Go ▪ Implemented a TCP server ▪ Used mongo-go-driver’s wireprotocol package ▪ Used mongo-go-driver's bson package
- 8. #MDBLocal Security Must have the same security as MongoDB. ▪ Users configured in Atlas ▪ Implemented MongoDB’s security model ▪ Authentication ▪ Authorization ▪ Require the use of TLS + SNI ▪ SNI = Server Name Indicator
- 9. #MDBLocal Behavior Must behave like MongoDB. ▪ Implemented commands for a read-only server ▪ Used the server’s aggregation engine
- 10. Customer’s Data
- 11. #MDBLocal Security: Customers Customers have complete control. ▪ Provide us with an IAM Role ▪ Configure your buckets ▪ Configure your users in Atlas
- 12. #MDBLocal Security: Atlas Atlas controls access to your data. ▪ Storage of IAM Role ▪ Temporary Credentials
- 13. #MDBLocal Configuration Customers control their data layout. ▪ Stores ▪ Databases, Collections ▪ DataSources CollectionCollection Store Store Database DataSource DataSource DataSource
- 14. #MDBLocal Configuration: File Formats Each file has a format. ▪ BSON (gzipped) ▪ JSON (gzipped) ▪ Avro (gzipped) ▪ CSV/TSV (gzipped) ▪ Parquet
- 15. MongoDB Atlas Data Lake UI
- 16. /archive/customers - a-m.json - n-z.json Configuration (S3 Bucket): ent-archive /archive/invoices - 2019 - 1.parquet - 2.parquet - 2018 - 1.parquet - 2017.json.gz - 2016.json.gz
- 17. s3 : { name: "ent-archive", bucket: "ent-archive", region: "us-west-2", prefix: "/archive/" } Configuration: Store
- 18. history: { customers: [{ store: "ent-archive", definition: "/customers/*" }], invoices: [{ store: "ent-archive", definition: "/invoices/{year int}/*" }, { store: "ent-archive", definition: "/invoices/{year int}" }] } Configuration: Data
- 19. history: { invoices: [{ store: "ent-archive", definition: "/invoices/{year int}/*" }, { store: "ent-archive", definition : "/invoices/{year int}” }, { store: "atlas", cluster: "my-cluster", db: "customers", collection: "invoices" }] } Configuration: Data (Future)
- 20. Queries
- 21. #MDBLocal Processing MQL Distributed MQL ▪ Parse ▪ Parallelize ▪ Distribute
- 23. { $match: { year: { $gt: 2000 } } } { $limit: 10 } Query Example: $limit Map: { $match: { year: { $gt: 2000 } } } { $limit: 10 } Reduce: { $limit: 10 }
- 24. { $group: { _id: "$year", totalAvg: { $avg: "amount" } } } Query Example: $group Map: { $group: { _id: "$year", totalAvg_sum: { $sum: "amount" }, totalAvg_count: { $sum: 1 } } } Reduce: { $group: { _id: "$_id", totalAvg_sum: { $sum: "$totalAvg_sum" }, totalAvg_count: { $sum: "$totalAvg_count" } } } Finalize: { $project: { _id: "$_id", totalAvg: { $divide: ["$totalAvg_sum", "$totalAvg_count"] } } }
- 25. DEMO Demo
- 26. Future
- 27. #MDBLocal Future More supported MongoDB operators. ▪ $out ▪ $merge ▪ $graphLookup ▪ Geo operators ▪ Full Text Search
- 29. #MDBLocal Future More File Formats! ▪ ORC ▪ Excel ▪ PDF
- 30. #MDBLocal Future Integrations! ▪ Atlas ▪ Microsoft Azure ▪ Google Cloud
- 31. #MDBLocal Hiring Lots to do! ▪ mongodb.com/careers
- 32. THANK YOU
- 33. #MDBlocal MongoDB Atlas Data Lake Technical Deep Dive [DEV] James Osgood https://www.surveymonkey.com/r/KQZZB8N