MongoDB .local Munich 2019: MongoDB Atlas Data Lake Technical Deep Dive
1 like282 views
The document discusses MongoDB Atlas Data Lake, emphasizing its design goals to efficiently manage and query vast amounts of data stored in various formats. It outlines the architecture, implementation requirements, and configuration process for creating an Atlas Data Lake, as well as how users can query and analyze long-term data using MongoDB Query Language (MQL). Future improvements and additional features are also mentioned, indicating a roadmap for enhancing the capability and performance of the data lake.
![Configuration: Databases & Collections
history: {
customers: [{
store: "ent-archive",
definition: "/customers/*"
}],
invoices: [{
store: "ent-archive",
definition: "/invoices/{year int}/*"
}, {
store: "ent-archive",
definition: "/invoices/{year int}.json.gz"
}]
}](https://image.slidesharecdn.com/905amatlasdatalakeipv2-191021140743/85/MongoDB-local-Munich-2019-MongoDB-Atlas-Data-Lake-Technical-Deep-Dive-19-320.jpg)
![{ $group: { _id: "$year", totalAvg: { $avg: "amount" } } }
Query Example: $group
Map:
{ $group: { _id: "$year",
totalAvg_sum: { $sum: "$amount" },
totalAvg_count: { $sum: 1 }
} }
Reduce:
{ $group: { _id: "$_id",
totalAvg_sum: { $sum: "$totalAvg_sum" },
totalAvg_count: { $sum: "$totalAvg_count" }
} }
Finalize:
{ $project: { _id: "$_id", totalAvg: { $divide: ["$totalAvg_sum", "$totalAvg_count"] } } }](https://image.slidesharecdn.com/905amatlasdatalakeipv2-191021140743/85/MongoDB-local-Munich-2019-MongoDB-Atlas-Data-Lake-Technical-Deep-Dive-26-320.jpg)
MongoDB .local Munich 2019: MongoDB Atlas Data Lake Technical Deep Dive
- 2. #MDBLocal “To free the genius within everyone by making data stunningly easy to work with.”
- 3. #MDBLocal Welcome to the World of Atlas Data Lake
- 4. #MDBLocal Isabel Peters Senior Software Engineer, MongoDB Atlas Backup
- 5. #MDBLocal Why are we building this? “IDC predicts that by 2025 worldwide data will reach 175 Zettabytes and 49% of it will reside in the public cloud. “ VS
- 6. #MDBLocal Atlas Data Lake Technical Deep Dive 1. Design Goals and Requirements 2. Creating an Atlas Data Lake 3. Atlas Data Lake Architecture 4. Future improvements
- 7. Design Goals and Requirements
- 9. #MDBLocal MongoDB Wire Protocol Support Requirements 1) Look and act like MongoDB Solution Empty • Implement a TCP server in Go. • Used mongo-go-driver’s wireprotocol packagey • Used mongo-go-driver's bson package • Read only
- 10. #MDBLocal MongoDB Security Model Requirements 2) Access customer’s data securely. Solution Empty • Users configured in MongoDB Atlas • Same authentication and authorization • Configure buckets
- 11. #MDBLocal Scalable Processing Requirements 3) Handle long running queries over vast amounts of data using resources efficiently Solution Empty • Read-only commands • Use server’s aggregation engine • Distributed MQL processing • Intelligent file targeting
- 12. #MDBLocal Data Formats Requirements 4) Support a variety of data formats Solution Empty • Avro (gzipped) • Parquet • BSON/ JSON (gzipped) • CSV/TSV (gzipped)
- 13. #MDBLocal Atlas Data Lake Features Multiple data formats Scalable MongoDB Query Language Serverless On Demand Integrated with Atlas
- 14. Creating your Atlas Data Lake
- 15. Files in S3 bucket: ent-archive /archive/customers - a-m.json - n-z.json /archive/invoices - 2019 - 1.parquet - 2.parquet - 2018 - 1.parquet - 2017.json.gz - 2016.json.gz
- 16. #MDBLocal You control your data layout Stores Empty Databases Empty Collections Empty DataSources CollectionCollection Store Store Database DataSource DataSource DataSource
- 17. #MDBLocal Data Lake Configuration 1. Configure a new Data Lake in Atlas 2. Connect to your Data Lake 3. Configure your databases and collections 4. Query your Data Lake
- 18. Configuration: S3 Store s3: { name: "ent-archive", bucket: "ent-archive", region: ”us-east-1", prefix: "/archive/" }
- 19. Configuration: Databases & Collections history: { customers: [{ store: "ent-archive", definition: "/customers/*" }], invoices: [{ store: "ent-archive", definition: "/invoices/{year int}/*" }, { store: "ent-archive", definition: "/invoices/{year int}.json.gz" }] }
- 20. #MDBLocal Querying via MongoDB Atlas • Atlas users require readWriteAnyDatabase or readAnyDatabase roles. • Use MongoDB drivers/clients including the mongo shell and MongoDB Compass • Write queries in MongoDB Query Language (MQL)
- 21. Atlas Data Lake Architecture
- 22. #MDBLocal MQL à Distributed MQL Parse query Parallelize processing Distribute workload
- 23. #MDBLocal Atlas Data Lake Architecture Atlas Control Control Plane Compute Plane Data Plane DataLake Frontend DataLake Agent Load Balancer Load Balancer DataLake Frontend DataLake Agent Load Balancer Load Balancer DataLake Frontend DataLake Agent Load Balancer Load Balancer
- 25. { $match: { year: { $gt: 2000 } } } { $limit: 10 } Query Example: $limit Map: { $match: { year: { $gt: 2000 } } } { $limit: 10 } Reduce: { $limit: 10 }
- 26. { $group: { _id: "$year", totalAvg: { $avg: "amount" } } } Query Example: $group Map: { $group: { _id: "$year", totalAvg_sum: { $sum: "$amount" }, totalAvg_count: { $sum: 1 } } } Reduce: { $group: { _id: "$_id", totalAvg_sum: { $sum: "$totalAvg_sum" }, totalAvg_count: { $sum: "$totalAvg_count" } } } Finalize: { $project: { _id: "$_id", totalAvg: { $divide: ["$totalAvg_sum", "$totalAvg_count"] } } }
- 28. #MDBLocal On the roadmap … MongoDB Operators $out $merge $graphLookup Performance • Aggregation • Indexes • Statistics over data File FormatsIntegrations
- 29. Summary
- 30. #MDBLocal Atlas Data Lake is the best way to: Access long-term data in multiple formats Query long-term data using MQL Analyse long-term data on demand
- 31. #MDBLocal Give it a try - Create your own Atlas Data Lake!
- 32. THANK YOU