SlideShare a Scribd company logo

ETL for Pros: Getting Data Into MongoDB

Download as PPTX, PDF
MongoDB
MongoDB

This document summarizes a presentation on best practices for extracting, transforming, and loading (ETL) large amounts of data from relational databases into MongoDB documents. The presentation discusses common mistakes made in ETL processes, including making nested database queries, building documents within the database, and loading all data into memory at once. It then analyzes a case study involving importing order, item, and tracking data from relational tables into normalized MongoDB documents.

Technology
1 of 77
Downloaded 15 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
FEBRUARY 15, 2018 | BELL HARBOR #MDBlocal ETL for Pros Getting Data into MongoDB
#MDBlocal Principal Consulting Engineer André Spiegel MongoDB @drmirror
#MDBlocal Remember this?
#MDBlocal At some point, most applications need to batch-load large amounts of data • billions of documents • huge initial load • daily updates Sound familiar?
#MDBlocal Using MongoDB properly means complex documents Sound familiar? { "_id" : "admin.mongo_dba", "user" : "mongo_dba", "db" : "admin", "roles" : [ { "role" : "root", "db" : "admin" }, { "role" : "restore", "db" : "admin" } ] } [ { "$sort" : { "st": 1 } }, { "$group" : { "_id" : "$st", "start" : { "$first" : "$ts" }, "end" : { "$last" : "$ts" } } } ]
#MDBlocal How do I create these documents from relational tables? Sound familiar?
#MDBlocal How do I do it fast? Sound familiar? Image: Julian Lim
#MDBlocal I've done this for a few years I've seen people do it We all make the same mistakes Let's understand them and come up with something better
#MDBlocal Case Study
ORDERS TRACKING ITEMS ID FIRST_NAME LAST_NAME SHIPPING_ADDRESS 1 James Bond Nassau, Bahamas, US 2 Ernst Blofeldt Caracas, Venezuela ID ORDER_ID QTY DESCRIPTION PRICE 1 1 1 Aston Martin 120,000 2 1 1 Dinner Jacket 4,000 3 1 3 Champagne Veuve-Cliquot 200 4 2 100 Cat Food 1 5 2 1 Launch Pad 1,000,000 ORDER_ID TIMESTAMP STATUS 1 1985-04-30 09:48:00 ORDERED 2 1985-04-23 01:30:22 ORDERED 2 1985-04-25 08:30:00 SHIPPED 2 1985-05-14 21:37:00 DELIVERED
ORDERS TRACKING ITEMS ID FIRST_NAME LAST_NAME SHIPPING_ADDRESS 1 James Bond Nassau, Bahamas, US 2 Ernst Blofeldt Caracas, Venezuela ID ORDER_ID QTY DESCRIPTION PRICE 1 1 1 Aston Martin 120,000 2 1 1 Dinner Jacket 4,000 3 1 3 Champagne Veuve-Cliquot 200 4 2 100 Cat Food 1 5 2 1 Launch Pad 1,000,000 ORDER_ID TIMESTAMP STATUS 1 1985-04-30 09:48:00 ORDERED 2 1985-04-23 01:30:22 ORDERED 2 1985-04-25 08:30:00 SHIPPED 2 1985-05-14 21:37:00 DELIVERED
ORDERS TRACKING ITEMS ID FIRST_NAME LAST_NAME SHIPPING_ADDRESS 1 James Bond Nassau, Bahamas, US 2 Ernst Blofeldt Caracas, Venezuela ID ORDER_ID QTY DESCRIPTION PRICE 1 1 1 Aston Martin 120,000 2 1 1 Dinner Jacket 4,000 3 1 3 Champagne Veuve-Cliquot 200 4 2 100 Cat Food 1 5 2 1 Launch Pad 1,000,000 ORDER_ID TIMESTAMP STATUS 1 1985-04-30 09:48:00 ORDERED 2 1985-04-23 01:30:22 ORDERED 2 1985-04-25 08:30:00 SHIPPED 2 1985-05-14 21:37:00 DELIVERED
{ "first_name" : "James", "last_name" : "Bond", "address" : "Nassau, Bahamas, US", "items" : [ { "qty": 1, "description" : "Aston Martin", "price" : 120000 }, { "qty": 1, "description" : "Dinner Jacket", "price" : 4000 }, { "qty": 3, "description" : "Champagne Veuve-Cliquot", "price": 200 } ], "tracking" : [ { "timestamp" : "1985-04-30 09:48:00", "status": "ORDERED" } ] }
{ "first_name" : "James", "last_name" : "Bond", "address" : "Nassau, Bahamas, US", "items" : [ { "qty": 1, "description" : "Aston Martin", "price" : 120000 }, { "qty": 1, "description" : "Dinner Jacket", "price" : 4000 }, { "qty": 3, "description" : "Champagne Veuve-Cliquot", "price": 200 } ], "tracking" : [ { "timestamp" : "1985-04-30 09:48:00", "status": "ORDERED" } ] }
{ "first_name" : "James", "last_name" : "Bond", "address" : "Nassau, Bahamas, US", "items" : [ { "qty": 1, "description" : "Aston Martin", "price" : 120000 }, { "qty": 1, "description" : "Dinner Jacket", "price" : 4000 }, { "qty": 3, "description" : "Champagne Veuve-Cliquot", "price": 200 } ], "tracking" : [ { "timestamp" : "1985-04-30 09:48:00", "status": "ORDERED" } ] }
{ "first_name" : "James", "last_name" : "Bond", "address" : "Nassau, Bahamas, US", "items" : [ { "qty": 1, "description" : "Aston Martin", "price" : 120000 }, { "qty": 1, "description" : "Dinner Jacket", "price" : 4000 }, { "qty": 3, "description" : "Champagne Veuve-Cliquot", "price": 200 } ], "tracking" : [ { "timestamp" : "1985-04-30 09:48:00", "status": "ORDERED" } ] }
#MDBlocal ETL Tools: Talend, Pentaho, Informatica, ... • Gretchen's Question: How do you handle arrays? How do I get from relational to JSON?
#MDBlocal WYOC (Write Your Own Code) • More challenging, but you've got ultimate control How do I get from relational to JSON?
#MDBlocal • Any operation in the CPU is on the order of nanoseconds: 0.000 000 001s • typically tens of nanoseconds per high-level operation • Any roundtrip to the database is on the order of milliseconds: 0.001s • typically just under 1 millisecond at the minimum • mostly due to network protocol stack latency • faster networks don't help • in-memory storage does not help Orders of Magnitude
#MDBlocal A Gallery Of Mistakes
ORDERS TRACKING ITEMS ID FIRST_NAME LAST_NAME SHIPPING_ADDRESS 1 James Bond Nassau, Bahamas, US 2 Ernst Blofeldt Caracas, Venezuela ID ORDER_ID QTY DESCRIPTION PRICE 1 1 1 Aston Martin 120,000 2 1 1 Dinner Jacket 4,000 3 1 3 Champagne Veuve-Cliquot 200 4 2 100 Cat Food 1 5 2 1 Launch Pad 1,000,000 ORDER_ID TIMESTAMP STATUS 1 1985-04-30 09:48:00 ORDERED 2 1985-04-23 01:30:22 ORDERED 2 1985-04-25 08:30:00 SHIPPED 2 1985-05-14 21:37:00 DELIVERED
#MDBlocal Mistake #1 – Nested queries for x in SELECT * FROM ORDERS doc = { "first_name" : x.first_name, "last_name" : x.last_name, "address" : x.address, "items" : [], "tracking" : [] } for y in SELECT * FROM ITEMS WHERE ORDER_ID = x.order_id doc.items.push (y) for z in SELECT * FROM TRACKING WHERE ORDER_ID = x.order_id doc.tracking.push (y) mongodb.insert (doc)
#MDBlocal Mistake #1 – Nested queries for x in SELECT * FROM ORDERS doc = { "first_name" : x.first_name, "last_name" : x.last_name, "address" : x.address, "items" : [], "tracking" : [] } for y in SELECT * FROM ITEMS WHERE ORDER_ID = x.order_id doc.items.push (y) for z in SELECT * FROM TRACKING WHERE ORDER_ID = x.order_id doc.tracking.push (y) mongodb.insert (doc)
#MDBlocal Mistake #1 – Nested queries for x in SELECT * FROM ORDERS doc = { "first_name" : x.first_name, "last_name" : x.last_name, "address" : x.address, "items" : [], "tracking" : [] } for y in SELECT * FROM ITEMS WHERE ORDER_ID = x.order_id doc.items.push (y) for z in SELECT * FROM TRACKING WHERE ORDER_ID = x.order_id doc.tracking.push (y) mongodb.insert (doc)
#MDBlocal Mistake #1 – Nested queries for x in SELECT * FROM ORDERS doc = { "first_name" : x.first_name, "last_name" : x.last_name, "address" : x.address, "items" : [], "tracking" : [] } for y in SELECT * FROM ITEMS WHERE ORDER_ID = x.order_id doc.items.push (y) for z in SELECT * FROM TRACKING WHERE ORDER_ID = x.order_id doc.tracking.push (y) mongodb.insert (doc)
#MDBlocal Mistake #1 – Nested queries for x in SELECT * FROM ORDERS doc = { "first_name" : x.first_name, "last_name" : x.last_name, "address" : x.address, "items" : [], "tracking" : [] } for y in SELECT * FROM ITEMS WHERE ORDER_ID = x.order_id doc.items.push (y) for z in SELECT * FROM TRACKING WHERE ORDER_ID = x.order_id doc.tracking.push (y) mongodb.insert (doc)
#MDBlocal Mistake #1 – Nested queries for x in SELECT * FROM ORDERS doc = { "first_name" : x.first_name, "last_name" : x.last_name, "address" : x.address, "items" : [], "tracking" : [] } for y in SELECT * FROM ITEMS WHERE ORDER_ID = x.order_id doc.items.push (y) for z in SELECT * FROM TRACKING WHERE ORDER_ID = x.order_id doc.tracking.push (y) mongodb.insert (doc)
#MDBlocal Mistake #1 – Nested queries for x in SELECT * FROM ORDERS doc = { "first_name" : x.first_name, "last_name" : x.last_name, "address" : x.address, "items" : [], "tracking" : [] } for y in SELECT * FROM ITEMS WHERE ORDER_ID = x.order_id doc.items.push (y) for z in SELECT * FROM TRACKING WHERE ORDER_ID = x.order_id doc.tracking.push (y) mongodb.insert (doc)
#MDBlocal Results • 1 million orders • 10 million line items • 3 million tracking states • MySQL (local) to MongoDB (local) • Python
#MDBlocal Fan-In and Fan-out ETL Job Number of Database Operations per MongoDB Document 1/n + 2 1
#MDBlocal Mistake #2 – Build documents in DB for x in SELECT * FROM ORDERS doc = { "first_name" : x.first_name, "last_name" : x.last_name, "address" : x.address, "items" : [], "tracking" : [] } mongodb.insert (doc) for y in SELECT * FROM ITEMS mongodb.update ({"_id" : y.order_id}, {"$push" : {"items" : y}}) for z in SELECT * FROM TRACKING mongodb.update ({"_id" : z.order_id}, {"$push" : {"tracking" : z}})
#MDBlocal Mistake #2 – Build documents in DB for x in SELECT * FROM ORDERS doc = { "first_name" : x.first_name, "last_name" : x.last_name, "address" : x.address, "items" : [], "tracking" : [] } mongodb.insert (doc) for y in SELECT * FROM ITEMS mongodb.update ({"_id" : y.order_id}, {"$push" : {"items" : y}}) for z in SELECT * FROM TRACKING mongodb.update ({"_id" : z.order_id}, {"$push" : {"tracking" : z}})
#MDBlocal Mistake #2 – Build documents in DB for x in SELECT * FROM ORDERS doc = { "first_name" : x.first_name, "last_name" : x.last_name, "address" : x.address, "items" : [], "tracking" : [] } mongodb.insert (doc) for y in SELECT * FROM ITEMS mongodb.update ({"_id" : y.order_id}, {"$push" : {"items" : y}}) for z in SELECT * FROM TRACKING mongodb.update ({"_id" : z.order_id}, {"$push" : {"tracking" : z}})
#MDBlocal Mistake #2 – Build documents in DB for x in SELECT * FROM ORDERS doc = { "first_name" : x.first_name, "last_name" : x.last_name, "address" : x.address, "items" : [], "tracking" : [] } mongodb.insert (doc) for y in SELECT * FROM ITEMS mongodb.update ({"_id" : y.order_id}, {"$push" : {"items" : y}}) for z in SELECT * FROM TRACKING mongodb.update ({"_id" : z.order_id}, {"$push" : {"tracking" : z}})
#MDBlocal Mistake #2 – Build documents in DB for x in SELECT * FROM ORDERS doc = { "first_name" : x.first_name, "last_name" : x.last_name, "address" : x.address, "items" : [], "tracking" : [] } mongodb.insert (doc) for y in SELECT * FROM ITEMS mongodb.update ({"_id" : y.order_id}, {"$push" : {"items" : y}}) for z in SELECT * FROM TRACKING mongodb.update ({"_id" : z.order_id}, {"$push" : {"tracking" : z}})
#MDBlocal Mistake #2 – Build documents in DB for x in SELECT * FROM ORDERS doc = { "first_name" : x.first_name, "last_name" : x.last_name, "address" : x.address, "items" : [], "tracking" : [] } mongodb.insert (doc) for y in SELECT * FROM ITEMS mongodb.update ({"_id" : y.order_id}, {"$push" : {"items" : y}}) for z in SELECT * FROM TRACKING mongodb.update ({"_id" : z.order_id}, {"$push" : {"tracking" : z}})
#MDBlocal Mistake #2 – Build documents in DB for x in SELECT * FROM ORDERS doc = { "first_name" : x.first_name, "last_name" : x.last_name, "address" : x.address, "items" : [], "tracking" : [] } mongodb.insert (doc) for y in SELECT * FROM ITEMS mongodb.update ({"_id" : y.order_id}, {"$push" : {"items" : y}}) for z in SELECT * FROM TRACKING mongodb.update ({"_id" : z.order_id}, {"$push" : {"tracking" : z}})
#MDBlocal Fan-In and Fan-out ETL Job Number of Database Operations per MongoDB Document 3/ n 1 + p + q
#MDBlocal Results
#MDBlocal Mistake #3 – Load it all into memory db_items = SELECT * FROM ITEMS db_tracking = SELECT * FROM TRACKING for x in SELECT * FROM ORDERS doc = { "first_name" : x.first_name, "last_name" : x.last_name, "address" : x.address, "items" : [], "tracking" : [] } doc.items.pushAll (db_items.getAll(x.order_id)) doc.tracking.pushAll (db_tracking.getAll(x.order_id)) mongodb.insert (doc)
#MDBlocal Mistake #3 – Load it all into memory db_items = SELECT * FROM ITEMS db_tracking = SELECT * FROM TRACKING for x in SELECT * FROM ORDERS doc = { "first_name" : x.first_name, "last_name" : x.last_name, "address" : x.address, "items" : [], "tracking" : [] } doc.items.pushAll (db_items.getAll(x.order_id)) doc.tracking.pushAll (db_tracking.getAll(x.order_id)) mongodb.insert (doc)
#MDBlocal Mistake #3 – Load it all into memory db_items = SELECT * FROM ITEMS db_tracking = SELECT * FROM TRACKING for x in SELECT * FROM ORDERS doc = { "first_name" : x.first_name, "last_name" : x.last_name, "address" : x.address, "items" : [], "tracking" : [] } doc.items.pushAll (db_items.getAll(x.order_id)) doc.tracking.pushAll (db_tracking.getAll(x.order_id)) mongodb.insert (doc)
#MDBlocal Mistake #3 – Load it all into memory db_items = SELECT * FROM ITEMS db_tracking = SELECT * FROM TRACKING for x in SELECT * FROM ORDERS doc = { "first_name" : x.first_name, "last_name" : x.last_name, "address" : x.address, "items" : [], "tracking" : [] } doc.items.pushAll (db_items.getAll(x.order_id)) doc.tracking.pushAll (db_tracking.getAll(x.order_id)) mongodb.insert (doc)
#MDBlocal Mistake #3 – Load it all into memory db_items = SELECT * FROM ITEMS db_tracking = SELECT * FROM TRACKING for x in SELECT * FROM ORDERS doc = { "first_name" : x.first_name, "last_name" : x.last_name, "address" : x.address, "items" : [], "tracking" : [] } doc.items.pushAll (db_items.getAll(x.order_id)) doc.tracking.pushAll (db_tracking.getAll(x.order_id)) mongodb.insert (doc)
#MDBlocal Mistake #3 – Load it all into memory db_items = SELECT * FROM ITEMS db_tracking = SELECT * FROM TRACKING for x in SELECT * FROM ORDERS doc = { "first_name" : x.first_name, "last_name" : x.last_name, "address" : x.address, "items" : [], "tracking" : [] } doc.items.pushAll (db_items.getAll(x.order_id)) doc.tracking.pushAll (db_tracking.getAll(x.order_id)) mongodb.insert (doc)
#MDBlocal Fan-In and Fan-out ETL Job Number of Database Operations per MongoDB Document 3/ n 1
#MDBlocal Results
#MDBlocal Getting it right: Co-Iteration
ORDERS TRACKING ITEMS ID FIRST_NAME LAST_NAME SHIPPING_ADDRESS 1 James Bond Nassau, Bahamas, US 2 Ernst Blofeldt Caracas, Venezuela ID ORDER_ID QTY DESCRIPTION PRICE 1 1 1 Aston Martin 120,000 2 1 1 Dinner Jacket 4,000 3 1 3 Champagne Veuve-Cliquot 200 4 2 100 Cat Food 1 5 2 1 Launch Pad 1,000,000 ORDER_ID TIMESTAMP STATUS 1 1985-04-30 09:48:00 ORDERED 2 1985-04-23 01:30:22 ORDERED 2 1985-04-25 08:30:00 SHIPPED 2 1985-05-14 21:37:00 DELIVERED
ORDERS TRACKING ITEMS ID FIRST_NAME LAST_NAME SHIPPING_ADDRESS 1 James Bond Nassau, Bahamas, US 2 Ernst Blofeldt Caracas, Venezuela ID ORDER_ID QTY DESCRIPTION PRICE 1 1 1 Aston Martin 120,000 2 1 1 Dinner Jacket 4,000 3 1 3 Champagne Veuve-Cliquot 200 4 2 100 Cat Food 1 5 2 1 Launch Pad 1,000,000 ORDER_ID TIMESTAMP STATUS 1 1985-04-30 09:48:00 ORDERED 2 1985-04-23 01:30:22 ORDERED 2 1985-04-25 08:30:00 SHIPPED 2 1985-05-14 21:37:00 DELIVERED
ORDERS TRACKING ITEMS ID FIRST_NAME LAST_NAME SHIPPING_ADDRESS 1 James Bond Nassau, Bahamas, US 2 Ernst Blofeldt Caracas, Venezuela ID ORDER_ID QTY DESCRIPTION PRICE 1 1 1 Aston Martin 120,000 2 1 1 Dinner Jacket 4,000 3 1 3 Champagne Veuve-Cliquot 200 4 2 100 Cat Food 1 5 2 1 Launch Pad 1,000,000 ORDER_ID TIMESTAMP STATUS 1 1985-04-30 09:48:00 ORDERED 2 1985-04-23 01:30:22 ORDERED 2 1985-04-25 08:30:00 SHIPPED 2 1985-05-14 21:37:00 DELIVERED { "first_name" : "James", "last_name" : "Bond", "address" : "Nassau, Bahamas, US" }
ORDERS TRACKING ITEMS ID FIRST_NAME LAST_NAME SHIPPING_ADDRESS 1 James Bond Nassau, Bahamas, US 2 Ernst Blofeldt Caracas, Venezuela ID ORDER_ID QTY DESCRIPTION PRICE 1 1 1 Aston Martin 120,000 2 1 1 Dinner Jacket 4,000 3 1 3 Champagne Veuve-Cliquot 200 4 2 100 Cat Food 1 5 2 1 Launch Pad 1,000,000 ORDER_ID TIMESTAMP STATUS 1 1985-04-30 09:48:00 ORDERED 2 1985-04-23 01:30:22 ORDERED 2 1985-04-25 08:30:00 SHIPPED 2 1985-05-14 21:37:00 DELIVERED { "first_name" : "James", "last_name" : "Bond", "address" : "Nassau, Bahamas, US", "items" : [ { ..., "description" : "Aston Martin", ... } ] }
ORDERS TRACKING ITEMS ID FIRST_NAME LAST_NAME SHIPPING_ADDRESS 1 James Bond Nassau, Bahamas, US 2 Ernst Blofeldt Caracas, Venezuela ID ORDER_ID QTY DESCRIPTION PRICE 1 1 1 Aston Martin 120,000 2 1 1 Dinner Jacket 4,000 3 1 3 Champagne Veuve-Cliquot 200 4 2 100 Cat Food 1 5 2 1 Launch Pad 1,000,000 ORDER_ID TIMESTAMP STATUS 1 1985-04-30 09:48:00 ORDERED 2 1985-04-23 01:30:22 ORDERED 2 1985-04-25 08:30:00 SHIPPED 2 1985-05-14 21:37:00 DELIVERED { "first_name" : "James", "last_name" : "Bond", "address" : "Nassau, Bahamas, US", "items" : [ { ..., "description" : "Aston Martin", ... }, { ..., "description" : "Dinner Jacket", ... } ] }
ORDERS TRACKING ITEMS ID FIRST_NAME LAST_NAME SHIPPING_ADDRESS 1 James Bond Nassau, Bahamas, US 2 Ernst Blofeldt Caracas, Venezuela ID ORDER_ID QTY DESCRIPTION PRICE 1 1 1 Aston Martin 120,000 2 1 1 Dinner Jacket 4,000 3 1 3 Champagne Veuve-Cliquot 200 4 2 100 Cat Food 1 5 2 1 Launch Pad 1,000,000 ORDER_ID TIMESTAMP STATUS 1 1985-04-30 09:48:00 ORDERED 2 1985-04-23 01:30:22 ORDERED 2 1985-04-25 08:30:00 SHIPPED 2 1985-05-14 21:37:00 DELIVERED { "first_name" : "James", "last_name" : "Bond", "address" : "Nassau, Bahamas, US", "items" : [ { ..., "description" : "Aston Martin", ... }, { ..., "description" : "Dinner Jacket", ... }, { ..., "description" : "Champagne...", ... } ] }
ORDERS TRACKING ITEMS ID FIRST_NAME LAST_NAME SHIPPING_ADDRESS 1 James Bond Nassau, Bahamas, US 2 Ernst Blofeldt Caracas, Venezuela ID ORDER_ID QTY DESCRIPTION PRICE 1 1 1 Aston Martin 120,000 2 1 1 Dinner Jacket 4,000 3 1 3 Champagne Veuve-Cliquot 200 4 2 100 Cat Food 1 5 2 1 Launch Pad 1,000,000 ORDER_ID TIMESTAMP STATUS 1 1985-04-30 09:48:00 ORDERED 2 1985-04-23 01:30:22 ORDERED 2 1985-04-25 08:30:00 SHIPPED 2 1985-05-14 21:37:00 DELIVERED { "first_name" : "James", "last_name" : "Bond", "address" : "Nassau, Bahamas, US", "items" : [ { ..., "description" : "Aston Martin", ... }, { ..., "description" : "Dinner Jacket", ... }, { ..., "description" : "Champagne...", ... } ] }
ORDERS TRACKING ITEMS ID FIRST_NAME LAST_NAME SHIPPING_ADDRESS 1 James Bond Nassau, Bahamas, US 2 Ernst Blofeldt Caracas, Venezuela ID ORDER_ID QTY DESCRIPTION PRICE 1 1 1 Aston Martin 120,000 2 1 1 Dinner Jacket 4,000 3 1 3 Champagne Veuve-Cliquot 200 4 2 100 Cat Food 1 5 2 1 Launch Pad 1,000,000 ORDER_ID TIMESTAMP STATUS 1 1985-04-30 09:48:00 ORDERED 2 1985-04-23 01:30:22 ORDERED 2 1985-04-25 08:30:00 SHIPPED 2 1985-05-14 21:37:00 DELIVERED { "first_name" : "James", "last_name" : "Bond", "address" : "Nassau, Bahamas, US", "items" : [ { ..., "description" : "Aston Martin", ... }, { ..., "description" : "Dinner Jacket", ... }, { ..., "description" : "Champagne...", ... } ], "tracking" : [ { ... "1985-04-30 09:48:00", ... "ORDERED" } ] }
ORDERS TRACKING ITEMS ID FIRST_NAME LAST_NAME SHIPPING_ADDRESS 1 James Bond Nassau, Bahamas, US 2 Ernst Blofeldt Caracas, Venezuela ID ORDER_ID QTY DESCRIPTION PRICE 1 1 1 Aston Martin 120,000 2 1 1 Dinner Jacket 4,000 3 1 3 Champagne Veuve-Cliquot 200 4 2 100 Cat Food 1 5 2 1 Launch Pad 1,000,000 ORDER_ID TIMESTAMP STATUS 1 1985-04-30 09:48:00 ORDERED 2 1985-04-23 01:30:22 ORDERED 2 1985-04-25 08:30:00 SHIPPED 2 1985-05-14 21:37:00 DELIVERED { "first_name" : "James", "last_name" : "Bond", "address" : "Nassau, Bahamas, US", "items" : [ { ..., "description" : "Aston Martin", ... }, { ..., "description" : "Dinner Jacket", ... }, { ..., "description" : "Champagne...", ... } ], "tracking" : [ { ... "1985-04-30 09:48:00", ... "ORDERED" } ] }
ORDERS TRACKING ITEMS ID FIRST_NAME LAST_NAME SHIPPING_ADDRESS 1 James Bond Nassau, Bahamas, US 2 Ernst Blofeldt Caracas, Venezuela ID ORDER_ID QTY DESCRIPTION PRICE 1 1 1 Aston Martin 120,000 2 1 1 Dinner Jacket 4,000 3 1 3 Champagne Veuve-Cliquot 200 4 2 100 Cat Food 1 5 2 1 Launch Pad 1,000,000 ORDER_ID TIMESTAMP STATUS 1 1985-04-30 09:48:00 ORDERED 2 1985-04-23 01:30:22 ORDERED 2 1985-04-25 08:30:00 SHIPPED 2 1985-05-14 21:37:00 DELIVERED
ORDERS TRACKING ITEMS ID FIRST_NAME LAST_NAME SHIPPING_ADDRESS 1 James Bond Nassau, Bahamas, US 2 Ernst Blofeldt Caracas, Venezuela ID ORDER_ID QTY DESCRIPTION PRICE 1 1 1 Aston Martin 120,000 2 1 1 Dinner Jacket 4,000 3 1 3 Champagne Veuve-Cliquot 200 4 2 100 Cat Food 1 5 2 1 Launch Pad 1,000,000 ORDER_ID TIMESTAMP STATUS 1 1985-04-30 09:48:00 ORDERED 2 1985-04-23 01:30:22 ORDERED 2 1985-04-25 08:30:00 SHIPPED 2 1985-05-14 21:37:00 DELIVERED
ORDERS TRACKING ITEMS ID FIRST_NAME LAST_NAME SHIPPING_ADDRESS 1 James Bond Nassau, Bahamas, US 2 Ernst Blofeldt Caracas, Venezuela ID ORDER_ID QTY DESCRIPTION PRICE 1 1 1 Aston Martin 120,000 2 1 1 Dinner Jacket 4,000 3 1 3 Champagne Veuve-Cliquot 200 4 2 100 Cat Food 1 5 2 1 Launch Pad 1,000,000 ORDER_ID TIMESTAMP STATUS 1 1985-04-30 09:48:00 ORDERED 2 1985-04-23 01:30:22 ORDERED 2 1985-04-25 08:30:00 SHIPPED 2 1985-05-14 21:37:00 DELIVERED { "first_name" : "Ernst", "last_name" : "Blofeldt", "address" : "Caracas, Venezuela" }
ORDERS TRACKING ITEMS ID FIRST_NAME LAST_NAME SHIPPING_ADDRESS 1 James Bond Nassau, Bahamas, US 2 Ernst Blofeldt Caracas, Venezuela ID ORDER_ID QTY DESCRIPTION PRICE 1 1 1 Aston Martin 120,000 2 1 1 Dinner Jacket 4,000 3 1 3 Champagne Veuve-Cliquot 200 4 2 100 Cat Food 1 5 2 1 Launch Pad 1,000,000 ORDER_ID TIMESTAMP STATUS 1 1985-04-30 09:48:00 ORDERED 2 1985-04-23 01:30:22 ORDERED 2 1985-04-25 08:30:00 SHIPPED 2 1985-05-14 21:37:00 DELIVERED { "first_name" : "Ernst", "last_name" : "Blofeldt", "address" : "Caracas, Venezuela", "items" : [ { ..., "description" : "Cat Food", ... } ] }
ORDERS TRACKING ITEMS ID FIRST_NAME LAST_NAME SHIPPING_ADDRESS 1 James Bond Nassau, Bahamas, US 2 Ernst Blofeldt Caracas, Venezuela ID ORDER_ID QTY DESCRIPTION PRICE 1 1 1 Aston Martin 120,000 2 1 1 Dinner Jacket 4,000 3 1 3 Champagne Veuve-Cliquot 200 4 2 100 Cat Food 1 5 2 1 Launch Pad 1,000,000 ORDER_ID TIMESTAMP STATUS 1 1985-04-30 09:48:00 ORDERED 2 1985-04-23 01:30:22 ORDERED 2 1985-04-25 08:30:00 SHIPPED 2 1985-05-14 21:37:00 DELIVERED { "first_name" : "Ernst", "last_name" : "Blofeldt", "address" : "Caracas, Venezuela", "items" : [ { ..., "description" : "Cat Food", ... }, { ..., "description" : "Launch Pad", ... } ] }
ORDERS TRACKING ITEMS ID FIRST_NAME LAST_NAME SHIPPING_ADDRESS 1 James Bond Nassau, Bahamas, US 2 Ernst Blofeldt Caracas, Venezuela ID ORDER_ID QTY DESCRIPTION PRICE 1 1 1 Aston Martin 120,000 2 1 1 Dinner Jacket 4,000 3 1 3 Champagne Veuve-Cliquot 200 4 2 100 Cat Food 1 5 2 1 Launch Pad 1,000,000 ORDER_ID TIMESTAMP STATUS 1 1985-04-30 09:48:00 ORDERED 2 1985-04-23 01:30:22 ORDERED 2 1985-04-25 08:30:00 SHIPPED 2 1985-05-14 21:37:00 DELIVERED { "first_name" : "Ernst", "last_name" : "Blofeldt", "address" : "Caracas, Venezuela", "items" : [ { ..., "description" : "Cat Food", ... }, { ..., "description" : "Launch Pad", ... } ] }
ORDERS TRACKING ITEMS ID FIRST_NAME LAST_NAME SHIPPING_ADDRESS 1 James Bond Nassau, Bahamas, US 2 Ernst Blofeldt Caracas, Venezuela ID ORDER_ID QTY DESCRIPTION PRICE 1 1 1 Aston Martin 120,000 2 1 1 Dinner Jacket 4,000 3 1 3 Champagne Veuve-Cliquot 200 4 2 100 Cat Food 1 5 2 1 Launch Pad 1,000,000 ORDER_ID TIMESTAMP STATUS 1 1985-04-30 09:48:00 ORDERED 2 1985-04-23 01:30:22 ORDERED 2 1985-04-25 08:30:00 SHIPPED 2 1985-05-14 21:37:00 DELIVERED { "first_name" : "Ernst", "last_name" : "Blofeldt", "address" : "Caracas, Venezuela", "items" : [ { ..., "description" : "Cat Food", ... }, { ..., "description" : "Launch Pad", ... } ], "tracking" : [ { ... "1985-04-23 01:30:22", ... "ORDERED" } ] }
ORDERS TRACKING ITEMS ID FIRST_NAME LAST_NAME SHIPPING_ADDRESS 1 James Bond Nassau, Bahamas, US 2 Ernst Blofeldt Caracas, Venezuela ID ORDER_ID QTY DESCRIPTION PRICE 1 1 1 Aston Martin 120,000 2 1 1 Dinner Jacket 4,000 3 1 3 Champagne Veuve-Cliquot 200 4 2 100 Cat Food 1 5 2 1 Launch Pad 1,000,000 ORDER_ID TIMESTAMP STATUS 1 1985-04-30 09:48:00 ORDERED 2 1985-04-23 01:30:22 ORDERED 2 1985-04-25 08:30:00 SHIPPED 2 1985-05-14 21:37:00 DELIVERED { "first_name" : "Ernst", "last_name" : "Blofeldt", "address" : "Caracas, Venezuela", "items" : [ { ..., "description" : "Cat Food", ... }, { ..., "description" : "Launch Pad", ... } ], "tracking" : [ { ... "1985-04-23 01:30:22", ... "ORDERED" }, { ... "1985-04-25 08:30:00", ... "SHIPPED" } ] }
ORDERS TRACKING ITEMS ID FIRST_NAME LAST_NAME SHIPPING_ADDRESS 1 James Bond Nassau, Bahamas, US 2 Ernst Blofeldt Caracas, Venezuela ID ORDER_ID QTY DESCRIPTION PRICE 1 1 1 Aston Martin 120,000 2 1 1 Dinner Jacket 4,000 3 1 3 Champagne Veuve-Cliquot 200 4 2 100 Cat Food 1 5 2 1 Launch Pad 1,000,000 ORDER_ID TIMESTAMP STATUS 1 1985-04-30 09:48:00 ORDERED 2 1985-04-23 01:30:22 ORDERED 2 1985-04-25 08:30:00 SHIPPED 2 1985-05-14 21:37:00 DELIVERED { "first_name" : "Ernst", "last_name" : "Blofeldt", "address" : "Caracas, Venezuela", "items" : [ { ..., "description" : "Cat Food", ... }, { ..., "description" : "Launch Pad", ... } ], "tracking" : [ { ... "1985-04-23 01:30:22", ... "ORDERED" }, { ... "1985-04-25 08:30:00", ... "SHIPPED" }, { ... "1985-05-14 21:37:00", .. "DELIVERED" } ] }
ORDERS TRACKING ITEMS ID FIRST_NAME LAST_NAME SHIPPING_ADDRESS 1 James Bond Nassau, Bahamas, US 2 Ernst Blofeldt Caracas, Venezuela ID ORDER_ID QTY DESCRIPTION PRICE 1 1 1 Aston Martin 120,000 2 1 1 Dinner Jacket 4,000 3 1 3 Champagne Veuve-Cliquot 200 4 2 100 Cat Food 1 5 2 1 Launch Pad 1,000,000 ORDER_ID TIMESTAMP STATUS 1 1985-04-30 09:48:00 ORDERED 2 1985-04-23 01:30:22 ORDERED 2 1985-04-25 08:30:00 SHIPPED 2 1985-05-14 21:37:00 DELIVERED { "first_name" : "Ernst", "last_name" : "Blofeldt", "address" : "Caracas, Venezuela", "items" : [ { ..., "description" : "Cat Food", ... }, { ..., "description" : "Launch Pad", ... } ], "tracking" : [ { ... "1985-04-23 01:30:22", ... "ORDERED" }, { ... "1985-04-25 08:30:00", ... "SHIPPED" }, { ... "1985-05-14 21:37:00", .. "DELIVERED" } ] }
ORDERS TRACKING ITEMS ID FIRST_NAME LAST_NAME SHIPPING_ADDRESS 1 James Bond Nassau, Bahamas, US 2 Ernst Blofeldt Caracas, Venezuela ID ORDER_ID QTY DESCRIPTION PRICE 1 1 1 Aston Martin 120,000 2 1 1 Dinner Jacket 4,000 3 1 3 Champagne Veuve-Cliquot 200 4 2 100 Cat Food 1 5 2 1 Launch Pad 1,000,000 ORDER_ID TIMESTAMP STATUS 1 1985-04-30 09:48:00 ORDERED 2 1985-04-23 01:30:22 ORDERED 2 1985-04-25 08:30:00 SHIPPED 2 1985-05-14 21:37:00 DELIVERED Done!
#MDBlocal Results
#MDBlocal Fan-In and Fan-Out ETL Job Number of Database Operations per MongoDB Document 3/ n 1
#MDBlocal • Yes. Although not as straightforward as you might think. Did you just explain to me what a JOIN is? • No. Co-Iteration works from multiple data sources. NAME ITEM TRACKING James Bond Aston Martin ORDERED James Bond Aston Martin SHIPPED James Bond Dinner Jacket ORDERED James Bond Dinner Jacket SHIPPED James Bond Champagne ORDERED James Bond Champagne SHIPPED
#MDBlocal Oh, and one more thing…
#MDBlocal Threading and Batching batc h size thread s throug h put
#MDBlocal Fan-In and Fan-out ETL Job Number of Database Operations per MongoDB Document 3/ n 1/1000
#MDBlocal Results
#MDBlocal • Common Mistakes to Watch Out For • Nested Queries • Building Documents in the Database • Loading Everything into Memory • The Co-Iteration Pattern • Open All Tables at Once • Perform a Single Pass over Them • Build Documents as You Go Along • Don't Forget Batching and Threading Summary
#MDBlocal Thank you. github.com/drmirror/etlpro

More Related Content

What's hot (20)

PDF
MongoDB .local San Francisco 2020: Aggregation Pipeline Power++
MongoDB
 
PDF
MongoDB World 2019: Aggregation Pipeline Power++: How MongoDB 4.2 Pipeline Em...
MongoDB
 
PPTX
Aggregation Framework
MongoDB
 
PDF
MongoDB Aggregation Framework
Caserta
 
PPTX
Aggregation in MongoDB
Kishor Parkhe
 
PDF
Embedding a language into string interpolator
Michael Limansky
 
PDF
MongoDB .local Bengaluru 2019: Aggregation Pipeline Power++: How MongoDB 4.2 ...
MongoDB
 
KEY
MongoDB Aggregation Framework
Tyler Brock
 
PPTX
Webinar: Exploring the Aggregation Framework
MongoDB
 
PPTX
ETL for Pros: Getting Data Into MongoDB
MongoDB
 
PPTX
The Aggregation Framework
MongoDB
 
PPTX
The Aggregation Framework
MongoDB
 
PDF
MongoDB Europe 2016 - Debugging MongoDB Performance
MongoDB
 
PPTX
Agg framework selectgroup feb2015 v2
MongoDB
 
PDF
Aggregation Framework MongoDB Days Munich
Norberto Leite
 
PPTX
MongoDB Aggregation
Amit Ghosh
 
PPTX
Webinar: Strongly Typed Languages and Flexible Schemas
MongoDB
 
PPTX
MongoDB World 2016 : Advanced Aggregation
Joe Drumgoole
 
PDF
Strongly Typed Languages and Flexible Schemas
Norberto Leite
 
PPTX
ETL for Pros: Getting Data Into MongoDB
MongoDB
 
MongoDB .local San Francisco 2020: Aggregation Pipeline Power++
MongoDB
 
MongoDB World 2019: Aggregation Pipeline Power++: How MongoDB 4.2 Pipeline Em...
MongoDB
 
Aggregation Framework
MongoDB
 
MongoDB Aggregation Framework
Caserta
 
Aggregation in MongoDB
Kishor Parkhe
 
Embedding a language into string interpolator
Michael Limansky
 
MongoDB .local Bengaluru 2019: Aggregation Pipeline Power++: How MongoDB 4.2 ...
MongoDB
 
MongoDB Aggregation Framework
Tyler Brock
 
Webinar: Exploring the Aggregation Framework
MongoDB
 
ETL for Pros: Getting Data Into MongoDB
MongoDB
 
The Aggregation Framework
MongoDB
 
The Aggregation Framework
MongoDB
 
MongoDB Europe 2016 - Debugging MongoDB Performance
MongoDB
 
Agg framework selectgroup feb2015 v2
MongoDB
 
Aggregation Framework MongoDB Days Munich
Norberto Leite
 
MongoDB Aggregation
Amit Ghosh
 
Webinar: Strongly Typed Languages and Flexible Schemas
MongoDB
 
MongoDB World 2016 : Advanced Aggregation
Joe Drumgoole
 
Strongly Typed Languages and Flexible Schemas
Norberto Leite
 
ETL for Pros: Getting Data Into MongoDB
MongoDB
 

Similar to ETL for Pros: Getting Data Into MongoDB (20)

PDF
Lab pratico per la progettazione di soluzioni MongoDB in ambito Internet of T...
festival ICT 2016
 
PDF
MongoDB Solution for Internet of Things and Big Data
Stefano Dindo
 
PDF
Simplifying & accelerating application development with MongoDB's intelligent...
Maxime Beugnet
 
PDF
Webinar: Schema Patterns and Your Storage Engine
MongoDB
 
PDF
MongoDB Meetup
Maxime Beugnet
 
PPTX
[MongoDB.local Bengaluru 2018] Keynote
MongoDB
 
PPTX
MongoDB World 2018: Keynote
MongoDB
 
PDF
MongoDB in FS
MongoDB
 
PDF
MongoDB .local Chicago 2019: Practical Data Modeling for MongoDB: Tutorial
MongoDB
 
KEY
NOSQL101, Or: How I Learned To Stop Worrying And Love The Mongo!
Daniel Cousineau
 
PPTX
Webinar: Position and Trade Management with MongoDB
MongoDB
 
PPTX
Jumpstart: Introduction to MongoDB
MongoDB
 
PDF
MongoDB World 2019: Benchmarking Transactions: MongoDB Meets TPC-C
MongoDB
 
PDF
Online | MongoDB Atlas on GCP Workshop
Natasha Wilson
 
PPTX
Webinar: Scaling MongoDB
MongoDB
 
PPTX
Introduction to MongoDB at IGDTUW
Ankur Raina
 
PPTX
MongoDB Evenings Minneapolis: MongoDB is Cool But When Should I Use It?
MongoDB
 
PDF
Single View of the Customer
MongoDB
 
PDF
Confluent & MongoDB APAC Lunch & Learn
confluent
 
PPTX
Intro to MongoDB (Extended Session)
All Things Open
 
Lab pratico per la progettazione di soluzioni MongoDB in ambito Internet of T...
festival ICT 2016
 
MongoDB Solution for Internet of Things and Big Data
Stefano Dindo
 
Simplifying & accelerating application development with MongoDB's intelligent...
Maxime Beugnet
 
Webinar: Schema Patterns and Your Storage Engine
MongoDB
 
MongoDB Meetup
Maxime Beugnet
 
[MongoDB.local Bengaluru 2018] Keynote
MongoDB
 
MongoDB World 2018: Keynote
MongoDB
 
MongoDB in FS
MongoDB
 
MongoDB .local Chicago 2019: Practical Data Modeling for MongoDB: Tutorial
MongoDB
 
NOSQL101, Or: How I Learned To Stop Worrying And Love The Mongo!
Daniel Cousineau
 
Webinar: Position and Trade Management with MongoDB
MongoDB
 
Jumpstart: Introduction to MongoDB
MongoDB
 
MongoDB World 2019: Benchmarking Transactions: MongoDB Meets TPC-C
MongoDB
 
Online | MongoDB Atlas on GCP Workshop
Natasha Wilson
 
Webinar: Scaling MongoDB
MongoDB
 
Introduction to MongoDB at IGDTUW
Ankur Raina
 
MongoDB Evenings Minneapolis: MongoDB is Cool But When Should I Use It?
MongoDB
 
Single View of the Customer
MongoDB
 
Confluent & MongoDB APAC Lunch & Learn
confluent
 
Intro to MongoDB (Extended Session)
All Things Open
 
Ad

More from MongoDB (20)

PDF
MongoDB SoCal 2020: Migrate Anything* to MongoDB Atlas
MongoDB
 
PDF
MongoDB SoCal 2020: Go on a Data Safari with MongoDB Charts!
MongoDB
 
PDF
MongoDB SoCal 2020: Using MongoDB Services in Kubernetes: Any Platform, Devel...
MongoDB
 
PDF
MongoDB SoCal 2020: A Complete Methodology of Data Modeling for MongoDB
MongoDB
 
PDF
MongoDB SoCal 2020: From Pharmacist to Analyst: Leveraging MongoDB for Real-T...
MongoDB
 
PDF
MongoDB SoCal 2020: Best Practices for Working with IoT and Time-series Data
MongoDB
 
PDF
MongoDB SoCal 2020: MongoDB Atlas Jump Start
MongoDB
 
PDF
MongoDB .local San Francisco 2020: Powering the new age data demands [Infosys]
MongoDB
 
PDF
MongoDB .local San Francisco 2020: Using Client Side Encryption in MongoDB 4.2
MongoDB
 
PDF
MongoDB .local San Francisco 2020: Using MongoDB Services in Kubernetes: any ...
MongoDB
 
PDF
MongoDB .local San Francisco 2020: Go on a Data Safari with MongoDB Charts!
MongoDB
 
PDF
MongoDB .local San Francisco 2020: From SQL to NoSQL -- Changing Your Mindset
MongoDB
 
PDF
MongoDB .local San Francisco 2020: MongoDB Atlas Jumpstart
MongoDB
 
PDF
MongoDB .local San Francisco 2020: Tips and Tricks++ for Querying and Indexin...
MongoDB
 
PDF
MongoDB .local San Francisco 2020: A Complete Methodology of Data Modeling fo...
MongoDB
 
PDF
MongoDB .local San Francisco 2020: MongoDB Atlas Data Lake Technical Deep Dive
MongoDB
 
PDF
MongoDB .local San Francisco 2020: Developing Alexa Skills with MongoDB & Golang
MongoDB
 
PDF
MongoDB .local Paris 2020: Realm : l'ingrédient secret pour de meilleures app...
MongoDB
 
PDF
MongoDB .local Paris 2020: Upply @MongoDB : Upply : Quand le Machine Learning...
MongoDB
 
PDF
MongoDB .local Paris 2020: Les bonnes pratiques pour sécuriser MongoDB
MongoDB
 
MongoDB SoCal 2020: Migrate Anything* to MongoDB Atlas
MongoDB
 
MongoDB SoCal 2020: Go on a Data Safari with MongoDB Charts!
MongoDB
 
MongoDB SoCal 2020: Using MongoDB Services in Kubernetes: Any Platform, Devel...
MongoDB
 
MongoDB SoCal 2020: A Complete Methodology of Data Modeling for MongoDB
MongoDB
 
MongoDB SoCal 2020: From Pharmacist to Analyst: Leveraging MongoDB for Real-T...
MongoDB
 
MongoDB SoCal 2020: Best Practices for Working with IoT and Time-series Data
MongoDB
 
MongoDB SoCal 2020: MongoDB Atlas Jump Start
MongoDB
 
MongoDB .local San Francisco 2020: Powering the new age data demands [Infosys]
MongoDB
 
MongoDB .local San Francisco 2020: Using Client Side Encryption in MongoDB 4.2
MongoDB
 
MongoDB .local San Francisco 2020: Using MongoDB Services in Kubernetes: any ...
MongoDB
 
MongoDB .local San Francisco 2020: Go on a Data Safari with MongoDB Charts!
MongoDB
 
MongoDB .local San Francisco 2020: From SQL to NoSQL -- Changing Your Mindset
MongoDB
 
MongoDB .local San Francisco 2020: MongoDB Atlas Jumpstart
MongoDB
 
MongoDB .local San Francisco 2020: Tips and Tricks++ for Querying and Indexin...
MongoDB
 
MongoDB .local San Francisco 2020: A Complete Methodology of Data Modeling fo...
MongoDB
 
MongoDB .local San Francisco 2020: MongoDB Atlas Data Lake Technical Deep Dive
MongoDB
 
MongoDB .local San Francisco 2020: Developing Alexa Skills with MongoDB & Golang
MongoDB
 
MongoDB .local Paris 2020: Realm : l'ingrédient secret pour de meilleures app...
MongoDB
 
MongoDB .local Paris 2020: Upply @MongoDB : Upply : Quand le Machine Learning...
MongoDB
 
MongoDB .local Paris 2020: Les bonnes pratiques pour sécuriser MongoDB
MongoDB
 
Ad

Recently uploaded (20)

PDF
Transcript: New from BookNet Canada for 2025: BNC BiblioShare - Tech Forum 2025
BookNet Canada
 
PDF
From Code to Challenge: Crafting Skill-Based Games That Engage and Reward
aiyshauae
 
PDF
Presentation - Vibe Coding The Future of Tech
yanuarsinggih1
 
PDF
CIFDAQ Market Insights for July 7th 2025
CIFDAQ
 
PDF
CIFDAQ Token Spotlight for 9th July 2025
CIFDAQ
 
PDF
Exolore The Essential AI Tools in 2025.pdf
Srinivasan M
 
PDF
Smart Trailers 2025 Update with History and Overview
Paul Menig
 
PPTX
From Sci-Fi to Reality: Exploring AI Evolution
Svetlana Meissner
 
PDF
Achieving Consistent and Reliable AI Code Generation - Medusa AI
medusaaico
 
PDF
NewMind AI - Journal 100 Insights After The 100th Issue
NewMind AI
 
PDF
SWEBOK Guide and Software Services Engineering Education
Hironori Washizaki
 
PDF
Timothy Rottach - Ramp up on AI Use Cases, from Vector Search to AI Agents wi...
AWS Chicago
 
PDF
DevBcn - Building 10x Organizations Using Modern Productivity Metrics
Justin Reock
 
PPTX
WooCommerce Workshop: Bring Your Laptop
Laura Hartwig
 
PPTX
Q2 FY26 Tableau User Group Leader Quarterly Call
lward7
 
PDF
"Beyond English: Navigating the Challenges of Building a Ukrainian-language R...
Fwdays
 
PDF
Agentic AI lifecycle for Enterprise Hyper-Automation
Debmalya Biswas
 
PPTX
Webinar: Introduction to LF Energy EVerest
DanBrown980551
 
PDF
LLMs.txt: Easily Control How AI Crawls Your Site
Keploy
 
PDF
Newgen 2022-Forrester Newgen TEI_13 05 2022-The-Total-Economic-Impact-Newgen-...
darshakparmar
 
Transcript: New from BookNet Canada for 2025: BNC BiblioShare - Tech Forum 2025
BookNet Canada
 
From Code to Challenge: Crafting Skill-Based Games That Engage and Reward
aiyshauae
 
Presentation - Vibe Coding The Future of Tech
yanuarsinggih1
 
CIFDAQ Market Insights for July 7th 2025
CIFDAQ
 
CIFDAQ Token Spotlight for 9th July 2025
CIFDAQ
 
Exolore The Essential AI Tools in 2025.pdf
Srinivasan M
 
Smart Trailers 2025 Update with History and Overview
Paul Menig
 
From Sci-Fi to Reality: Exploring AI Evolution
Svetlana Meissner
 
Achieving Consistent and Reliable AI Code Generation - Medusa AI
medusaaico
 
NewMind AI - Journal 100 Insights After The 100th Issue
NewMind AI
 
SWEBOK Guide and Software Services Engineering Education
Hironori Washizaki
 
Timothy Rottach - Ramp up on AI Use Cases, from Vector Search to AI Agents wi...
AWS Chicago
 
DevBcn - Building 10x Organizations Using Modern Productivity Metrics
Justin Reock
 
WooCommerce Workshop: Bring Your Laptop
Laura Hartwig
 
Q2 FY26 Tableau User Group Leader Quarterly Call
lward7
 
"Beyond English: Navigating the Challenges of Building a Ukrainian-language R...
Fwdays
 
Agentic AI lifecycle for Enterprise Hyper-Automation
Debmalya Biswas
 
Webinar: Introduction to LF Energy EVerest
DanBrown980551
 
LLMs.txt: Easily Control How AI Crawls Your Site
Keploy
 
Newgen 2022-Forrester Newgen TEI_13 05 2022-The-Total-Economic-Impact-Newgen-...
darshakparmar
 

ETL for Pros: Getting Data Into MongoDB

  • 1. FEBRUARY 15, 2018 | BELL HARBOR #MDBlocal ETL for Pros Getting Data into MongoDB
  • 4. #MDBlocal At some point, most applications need to batch-load large amounts of data • billions of documents • huge initial load • daily updates Sound familiar?
  • 5. #MDBlocal Using MongoDB properly means complex documents Sound familiar? { "_id" : "admin.mongo_dba", "user" : "mongo_dba", "db" : "admin", "roles" : [ { "role" : "root", "db" : "admin" }, { "role" : "restore", "db" : "admin" } ] } [ { "$sort" : { "st": 1 } }, { "$group" : { "_id" : "$st", "start" : { "$first" : "$ts" }, "end" : { "$last" : "$ts" } } } ]
  • 6. #MDBlocal How do I create these documents from relational tables? Sound familiar?
  • 7. #MDBlocal How do I do it fast? Sound familiar? Image: Julian Lim
  • 8. #MDBlocal I've done this for a few years I've seen people do it We all make the same mistakes Let's understand them and come up with something better
  • 10. ORDERS TRACKING ITEMS ID FIRST_NAME LAST_NAME SHIPPING_ADDRESS 1 James Bond Nassau, Bahamas, US 2 Ernst Blofeldt Caracas, Venezuela ID ORDER_ID QTY DESCRIPTION PRICE 1 1 1 Aston Martin 120,000 2 1 1 Dinner Jacket 4,000 3 1 3 Champagne Veuve-Cliquot 200 4 2 100 Cat Food 1 5 2 1 Launch Pad 1,000,000 ORDER_ID TIMESTAMP STATUS 1 1985-04-30 09:48:00 ORDERED 2 1985-04-23 01:30:22 ORDERED 2 1985-04-25 08:30:00 SHIPPED 2 1985-05-14 21:37:00 DELIVERED
  • 11. ORDERS TRACKING ITEMS ID FIRST_NAME LAST_NAME SHIPPING_ADDRESS 1 James Bond Nassau, Bahamas, US 2 Ernst Blofeldt Caracas, Venezuela ID ORDER_ID QTY DESCRIPTION PRICE 1 1 1 Aston Martin 120,000 2 1 1 Dinner Jacket 4,000 3 1 3 Champagne Veuve-Cliquot 200 4 2 100 Cat Food 1 5 2 1 Launch Pad 1,000,000 ORDER_ID TIMESTAMP STATUS 1 1985-04-30 09:48:00 ORDERED 2 1985-04-23 01:30:22 ORDERED 2 1985-04-25 08:30:00 SHIPPED 2 1985-05-14 21:37:00 DELIVERED
  • 12. ORDERS TRACKING ITEMS ID FIRST_NAME LAST_NAME SHIPPING_ADDRESS 1 James Bond Nassau, Bahamas, US 2 Ernst Blofeldt Caracas, Venezuela ID ORDER_ID QTY DESCRIPTION PRICE 1 1 1 Aston Martin 120,000 2 1 1 Dinner Jacket 4,000 3 1 3 Champagne Veuve-Cliquot 200 4 2 100 Cat Food 1 5 2 1 Launch Pad 1,000,000 ORDER_ID TIMESTAMP STATUS 1 1985-04-30 09:48:00 ORDERED 2 1985-04-23 01:30:22 ORDERED 2 1985-04-25 08:30:00 SHIPPED 2 1985-05-14 21:37:00 DELIVERED
  • 13. { "first_name" : "James", "last_name" : "Bond", "address" : "Nassau, Bahamas, US", "items" : [ { "qty": 1, "description" : "Aston Martin", "price" : 120000 }, { "qty": 1, "description" : "Dinner Jacket", "price" : 4000 }, { "qty": 3, "description" : "Champagne Veuve-Cliquot", "price": 200 } ], "tracking" : [ { "timestamp" : "1985-04-30 09:48:00", "status": "ORDERED" } ] }
  • 14. { "first_name" : "James", "last_name" : "Bond", "address" : "Nassau, Bahamas, US", "items" : [ { "qty": 1, "description" : "Aston Martin", "price" : 120000 }, { "qty": 1, "description" : "Dinner Jacket", "price" : 4000 }, { "qty": 3, "description" : "Champagne Veuve-Cliquot", "price": 200 } ], "tracking" : [ { "timestamp" : "1985-04-30 09:48:00", "status": "ORDERED" } ] }
  • 15. { "first_name" : "James", "last_name" : "Bond", "address" : "Nassau, Bahamas, US", "items" : [ { "qty": 1, "description" : "Aston Martin", "price" : 120000 }, { "qty": 1, "description" : "Dinner Jacket", "price" : 4000 }, { "qty": 3, "description" : "Champagne Veuve-Cliquot", "price": 200 } ], "tracking" : [ { "timestamp" : "1985-04-30 09:48:00", "status": "ORDERED" } ] }
  • 16. { "first_name" : "James", "last_name" : "Bond", "address" : "Nassau, Bahamas, US", "items" : [ { "qty": 1, "description" : "Aston Martin", "price" : 120000 }, { "qty": 1, "description" : "Dinner Jacket", "price" : 4000 }, { "qty": 3, "description" : "Champagne Veuve-Cliquot", "price": 200 } ], "tracking" : [ { "timestamp" : "1985-04-30 09:48:00", "status": "ORDERED" } ] }
  • 17. #MDBlocal ETL Tools: Talend, Pentaho, Informatica, ... • Gretchen's Question: How do you handle arrays? How do I get from relational to JSON?
  • 18. #MDBlocal WYOC (Write Your Own Code) • More challenging, but you've got ultimate control How do I get from relational to JSON?
  • 19. #MDBlocal • Any operation in the CPU is on the order of nanoseconds: 0.000 000 001s • typically tens of nanoseconds per high-level operation • Any roundtrip to the database is on the order of milliseconds: 0.001s • typically just under 1 millisecond at the minimum • mostly due to network protocol stack latency • faster networks don't help • in-memory storage does not help Orders of Magnitude
  • 21. ORDERS TRACKING ITEMS ID FIRST_NAME LAST_NAME SHIPPING_ADDRESS 1 James Bond Nassau, Bahamas, US 2 Ernst Blofeldt Caracas, Venezuela ID ORDER_ID QTY DESCRIPTION PRICE 1 1 1 Aston Martin 120,000 2 1 1 Dinner Jacket 4,000 3 1 3 Champagne Veuve-Cliquot 200 4 2 100 Cat Food 1 5 2 1 Launch Pad 1,000,000 ORDER_ID TIMESTAMP STATUS 1 1985-04-30 09:48:00 ORDERED 2 1985-04-23 01:30:22 ORDERED 2 1985-04-25 08:30:00 SHIPPED 2 1985-05-14 21:37:00 DELIVERED
  • 22. #MDBlocal Mistake #1 – Nested queries for x in SELECT * FROM ORDERS doc = { "first_name" : x.first_name, "last_name" : x.last_name, "address" : x.address, "items" : [], "tracking" : [] } for y in SELECT * FROM ITEMS WHERE ORDER_ID = x.order_id doc.items.push (y) for z in SELECT * FROM TRACKING WHERE ORDER_ID = x.order_id doc.tracking.push (y) mongodb.insert (doc)
  • 23. #MDBlocal Mistake #1 – Nested queries for x in SELECT * FROM ORDERS doc = { "first_name" : x.first_name, "last_name" : x.last_name, "address" : x.address, "items" : [], "tracking" : [] } for y in SELECT * FROM ITEMS WHERE ORDER_ID = x.order_id doc.items.push (y) for z in SELECT * FROM TRACKING WHERE ORDER_ID = x.order_id doc.tracking.push (y) mongodb.insert (doc)
  • 24. #MDBlocal Mistake #1 – Nested queries for x in SELECT * FROM ORDERS doc = { "first_name" : x.first_name, "last_name" : x.last_name, "address" : x.address, "items" : [], "tracking" : [] } for y in SELECT * FROM ITEMS WHERE ORDER_ID = x.order_id doc.items.push (y) for z in SELECT * FROM TRACKING WHERE ORDER_ID = x.order_id doc.tracking.push (y) mongodb.insert (doc)
  • 25. #MDBlocal Mistake #1 – Nested queries for x in SELECT * FROM ORDERS doc = { "first_name" : x.first_name, "last_name" : x.last_name, "address" : x.address, "items" : [], "tracking" : [] } for y in SELECT * FROM ITEMS WHERE ORDER_ID = x.order_id doc.items.push (y) for z in SELECT * FROM TRACKING WHERE ORDER_ID = x.order_id doc.tracking.push (y) mongodb.insert (doc)
  • 26. #MDBlocal Mistake #1 – Nested queries for x in SELECT * FROM ORDERS doc = { "first_name" : x.first_name, "last_name" : x.last_name, "address" : x.address, "items" : [], "tracking" : [] } for y in SELECT * FROM ITEMS WHERE ORDER_ID = x.order_id doc.items.push (y) for z in SELECT * FROM TRACKING WHERE ORDER_ID = x.order_id doc.tracking.push (y) mongodb.insert (doc)
  • 27. #MDBlocal Mistake #1 – Nested queries for x in SELECT * FROM ORDERS doc = { "first_name" : x.first_name, "last_name" : x.last_name, "address" : x.address, "items" : [], "tracking" : [] } for y in SELECT * FROM ITEMS WHERE ORDER_ID = x.order_id doc.items.push (y) for z in SELECT * FROM TRACKING WHERE ORDER_ID = x.order_id doc.tracking.push (y) mongodb.insert (doc)
  • 28. #MDBlocal Mistake #1 – Nested queries for x in SELECT * FROM ORDERS doc = { "first_name" : x.first_name, "last_name" : x.last_name, "address" : x.address, "items" : [], "tracking" : [] } for y in SELECT * FROM ITEMS WHERE ORDER_ID = x.order_id doc.items.push (y) for z in SELECT * FROM TRACKING WHERE ORDER_ID = x.order_id doc.tracking.push (y) mongodb.insert (doc)
  • 29. #MDBlocal Results • 1 million orders • 10 million line items • 3 million tracking states • MySQL (local) to MongoDB (local) • Python
  • 30. #MDBlocal Fan-In and Fan-out ETL Job Number of Database Operations per MongoDB Document 1/n + 2 1
  • 31. #MDBlocal Mistake #2 – Build documents in DB for x in SELECT * FROM ORDERS doc = { "first_name" : x.first_name, "last_name" : x.last_name, "address" : x.address, "items" : [], "tracking" : [] } mongodb.insert (doc) for y in SELECT * FROM ITEMS mongodb.update ({"_id" : y.order_id}, {"$push" : {"items" : y}}) for z in SELECT * FROM TRACKING mongodb.update ({"_id" : z.order_id}, {"$push" : {"tracking" : z}})
  • 32. #MDBlocal Mistake #2 – Build documents in DB for x in SELECT * FROM ORDERS doc = { "first_name" : x.first_name, "last_name" : x.last_name, "address" : x.address, "items" : [], "tracking" : [] } mongodb.insert (doc) for y in SELECT * FROM ITEMS mongodb.update ({"_id" : y.order_id}, {"$push" : {"items" : y}}) for z in SELECT * FROM TRACKING mongodb.update ({"_id" : z.order_id}, {"$push" : {"tracking" : z}})
  • 33. #MDBlocal Mistake #2 – Build documents in DB for x in SELECT * FROM ORDERS doc = { "first_name" : x.first_name, "last_name" : x.last_name, "address" : x.address, "items" : [], "tracking" : [] } mongodb.insert (doc) for y in SELECT * FROM ITEMS mongodb.update ({"_id" : y.order_id}, {"$push" : {"items" : y}}) for z in SELECT * FROM TRACKING mongodb.update ({"_id" : z.order_id}, {"$push" : {"tracking" : z}})
  • 34. #MDBlocal Mistake #2 – Build documents in DB for x in SELECT * FROM ORDERS doc = { "first_name" : x.first_name, "last_name" : x.last_name, "address" : x.address, "items" : [], "tracking" : [] } mongodb.insert (doc) for y in SELECT * FROM ITEMS mongodb.update ({"_id" : y.order_id}, {"$push" : {"items" : y}}) for z in SELECT * FROM TRACKING mongodb.update ({"_id" : z.order_id}, {"$push" : {"tracking" : z}})
  • 35. #MDBlocal Mistake #2 – Build documents in DB for x in SELECT * FROM ORDERS doc = { "first_name" : x.first_name, "last_name" : x.last_name, "address" : x.address, "items" : [], "tracking" : [] } mongodb.insert (doc) for y in SELECT * FROM ITEMS mongodb.update ({"_id" : y.order_id}, {"$push" : {"items" : y}}) for z in SELECT * FROM TRACKING mongodb.update ({"_id" : z.order_id}, {"$push" : {"tracking" : z}})
  • 36. #MDBlocal Mistake #2 – Build documents in DB for x in SELECT * FROM ORDERS doc = { "first_name" : x.first_name, "last_name" : x.last_name, "address" : x.address, "items" : [], "tracking" : [] } mongodb.insert (doc) for y in SELECT * FROM ITEMS mongodb.update ({"_id" : y.order_id}, {"$push" : {"items" : y}}) for z in SELECT * FROM TRACKING mongodb.update ({"_id" : z.order_id}, {"$push" : {"tracking" : z}})
  • 37. #MDBlocal Mistake #2 – Build documents in DB for x in SELECT * FROM ORDERS doc = { "first_name" : x.first_name, "last_name" : x.last_name, "address" : x.address, "items" : [], "tracking" : [] } mongodb.insert (doc) for y in SELECT * FROM ITEMS mongodb.update ({"_id" : y.order_id}, {"$push" : {"items" : y}}) for z in SELECT * FROM TRACKING mongodb.update ({"_id" : z.order_id}, {"$push" : {"tracking" : z}})
  • 38. #MDBlocal Fan-In and Fan-out ETL Job Number of Database Operations per MongoDB Document 3/ n 1 + p + q
  • 40. #MDBlocal Mistake #3 – Load it all into memory db_items = SELECT * FROM ITEMS db_tracking = SELECT * FROM TRACKING for x in SELECT * FROM ORDERS doc = { "first_name" : x.first_name, "last_name" : x.last_name, "address" : x.address, "items" : [], "tracking" : [] } doc.items.pushAll (db_items.getAll(x.order_id)) doc.tracking.pushAll (db_tracking.getAll(x.order_id)) mongodb.insert (doc)
  • 41. #MDBlocal Mistake #3 – Load it all into memory db_items = SELECT * FROM ITEMS db_tracking = SELECT * FROM TRACKING for x in SELECT * FROM ORDERS doc = { "first_name" : x.first_name, "last_name" : x.last_name, "address" : x.address, "items" : [], "tracking" : [] } doc.items.pushAll (db_items.getAll(x.order_id)) doc.tracking.pushAll (db_tracking.getAll(x.order_id)) mongodb.insert (doc)
  • 42. #MDBlocal Mistake #3 – Load it all into memory db_items = SELECT * FROM ITEMS db_tracking = SELECT * FROM TRACKING for x in SELECT * FROM ORDERS doc = { "first_name" : x.first_name, "last_name" : x.last_name, "address" : x.address, "items" : [], "tracking" : [] } doc.items.pushAll (db_items.getAll(x.order_id)) doc.tracking.pushAll (db_tracking.getAll(x.order_id)) mongodb.insert (doc)
  • 43. #MDBlocal Mistake #3 – Load it all into memory db_items = SELECT * FROM ITEMS db_tracking = SELECT * FROM TRACKING for x in SELECT * FROM ORDERS doc = { "first_name" : x.first_name, "last_name" : x.last_name, "address" : x.address, "items" : [], "tracking" : [] } doc.items.pushAll (db_items.getAll(x.order_id)) doc.tracking.pushAll (db_tracking.getAll(x.order_id)) mongodb.insert (doc)
  • 44. #MDBlocal Mistake #3 – Load it all into memory db_items = SELECT * FROM ITEMS db_tracking = SELECT * FROM TRACKING for x in SELECT * FROM ORDERS doc = { "first_name" : x.first_name, "last_name" : x.last_name, "address" : x.address, "items" : [], "tracking" : [] } doc.items.pushAll (db_items.getAll(x.order_id)) doc.tracking.pushAll (db_tracking.getAll(x.order_id)) mongodb.insert (doc)
  • 45. #MDBlocal Mistake #3 – Load it all into memory db_items = SELECT * FROM ITEMS db_tracking = SELECT * FROM TRACKING for x in SELECT * FROM ORDERS doc = { "first_name" : x.first_name, "last_name" : x.last_name, "address" : x.address, "items" : [], "tracking" : [] } doc.items.pushAll (db_items.getAll(x.order_id)) doc.tracking.pushAll (db_tracking.getAll(x.order_id)) mongodb.insert (doc)
  • 46. #MDBlocal Fan-In and Fan-out ETL Job Number of Database Operations per MongoDB Document 3/ n 1
  • 49. ORDERS TRACKING ITEMS ID FIRST_NAME LAST_NAME SHIPPING_ADDRESS 1 James Bond Nassau, Bahamas, US 2 Ernst Blofeldt Caracas, Venezuela ID ORDER_ID QTY DESCRIPTION PRICE 1 1 1 Aston Martin 120,000 2 1 1 Dinner Jacket 4,000 3 1 3 Champagne Veuve-Cliquot 200 4 2 100 Cat Food 1 5 2 1 Launch Pad 1,000,000 ORDER_ID TIMESTAMP STATUS 1 1985-04-30 09:48:00 ORDERED 2 1985-04-23 01:30:22 ORDERED 2 1985-04-25 08:30:00 SHIPPED 2 1985-05-14 21:37:00 DELIVERED
  • 50. ORDERS TRACKING ITEMS ID FIRST_NAME LAST_NAME SHIPPING_ADDRESS 1 James Bond Nassau, Bahamas, US 2 Ernst Blofeldt Caracas, Venezuela ID ORDER_ID QTY DESCRIPTION PRICE 1 1 1 Aston Martin 120,000 2 1 1 Dinner Jacket 4,000 3 1 3 Champagne Veuve-Cliquot 200 4 2 100 Cat Food 1 5 2 1 Launch Pad 1,000,000 ORDER_ID TIMESTAMP STATUS 1 1985-04-30 09:48:00 ORDERED 2 1985-04-23 01:30:22 ORDERED 2 1985-04-25 08:30:00 SHIPPED 2 1985-05-14 21:37:00 DELIVERED
  • 51. ORDERS TRACKING ITEMS ID FIRST_NAME LAST_NAME SHIPPING_ADDRESS 1 James Bond Nassau, Bahamas, US 2 Ernst Blofeldt Caracas, Venezuela ID ORDER_ID QTY DESCRIPTION PRICE 1 1 1 Aston Martin 120,000 2 1 1 Dinner Jacket 4,000 3 1 3 Champagne Veuve-Cliquot 200 4 2 100 Cat Food 1 5 2 1 Launch Pad 1,000,000 ORDER_ID TIMESTAMP STATUS 1 1985-04-30 09:48:00 ORDERED 2 1985-04-23 01:30:22 ORDERED 2 1985-04-25 08:30:00 SHIPPED 2 1985-05-14 21:37:00 DELIVERED { "first_name" : "James", "last_name" : "Bond", "address" : "Nassau, Bahamas, US" }
  • 52. ORDERS TRACKING ITEMS ID FIRST_NAME LAST_NAME SHIPPING_ADDRESS 1 James Bond Nassau, Bahamas, US 2 Ernst Blofeldt Caracas, Venezuela ID ORDER_ID QTY DESCRIPTION PRICE 1 1 1 Aston Martin 120,000 2 1 1 Dinner Jacket 4,000 3 1 3 Champagne Veuve-Cliquot 200 4 2 100 Cat Food 1 5 2 1 Launch Pad 1,000,000 ORDER_ID TIMESTAMP STATUS 1 1985-04-30 09:48:00 ORDERED 2 1985-04-23 01:30:22 ORDERED 2 1985-04-25 08:30:00 SHIPPED 2 1985-05-14 21:37:00 DELIVERED { "first_name" : "James", "last_name" : "Bond", "address" : "Nassau, Bahamas, US", "items" : [ { ..., "description" : "Aston Martin", ... } ] }
  • 53. ORDERS TRACKING ITEMS ID FIRST_NAME LAST_NAME SHIPPING_ADDRESS 1 James Bond Nassau, Bahamas, US 2 Ernst Blofeldt Caracas, Venezuela ID ORDER_ID QTY DESCRIPTION PRICE 1 1 1 Aston Martin 120,000 2 1 1 Dinner Jacket 4,000 3 1 3 Champagne Veuve-Cliquot 200 4 2 100 Cat Food 1 5 2 1 Launch Pad 1,000,000 ORDER_ID TIMESTAMP STATUS 1 1985-04-30 09:48:00 ORDERED 2 1985-04-23 01:30:22 ORDERED 2 1985-04-25 08:30:00 SHIPPED 2 1985-05-14 21:37:00 DELIVERED { "first_name" : "James", "last_name" : "Bond", "address" : "Nassau, Bahamas, US", "items" : [ { ..., "description" : "Aston Martin", ... }, { ..., "description" : "Dinner Jacket", ... } ] }
  • 54. ORDERS TRACKING ITEMS ID FIRST_NAME LAST_NAME SHIPPING_ADDRESS 1 James Bond Nassau, Bahamas, US 2 Ernst Blofeldt Caracas, Venezuela ID ORDER_ID QTY DESCRIPTION PRICE 1 1 1 Aston Martin 120,000 2 1 1 Dinner Jacket 4,000 3 1 3 Champagne Veuve-Cliquot 200 4 2 100 Cat Food 1 5 2 1 Launch Pad 1,000,000 ORDER_ID TIMESTAMP STATUS 1 1985-04-30 09:48:00 ORDERED 2 1985-04-23 01:30:22 ORDERED 2 1985-04-25 08:30:00 SHIPPED 2 1985-05-14 21:37:00 DELIVERED { "first_name" : "James", "last_name" : "Bond", "address" : "Nassau, Bahamas, US", "items" : [ { ..., "description" : "Aston Martin", ... }, { ..., "description" : "Dinner Jacket", ... }, { ..., "description" : "Champagne...", ... } ] }
  • 55. ORDERS TRACKING ITEMS ID FIRST_NAME LAST_NAME SHIPPING_ADDRESS 1 James Bond Nassau, Bahamas, US 2 Ernst Blofeldt Caracas, Venezuela ID ORDER_ID QTY DESCRIPTION PRICE 1 1 1 Aston Martin 120,000 2 1 1 Dinner Jacket 4,000 3 1 3 Champagne Veuve-Cliquot 200 4 2 100 Cat Food 1 5 2 1 Launch Pad 1,000,000 ORDER_ID TIMESTAMP STATUS 1 1985-04-30 09:48:00 ORDERED 2 1985-04-23 01:30:22 ORDERED 2 1985-04-25 08:30:00 SHIPPED 2 1985-05-14 21:37:00 DELIVERED { "first_name" : "James", "last_name" : "Bond", "address" : "Nassau, Bahamas, US", "items" : [ { ..., "description" : "Aston Martin", ... }, { ..., "description" : "Dinner Jacket", ... }, { ..., "description" : "Champagne...", ... } ] }
  • 56. ORDERS TRACKING ITEMS ID FIRST_NAME LAST_NAME SHIPPING_ADDRESS 1 James Bond Nassau, Bahamas, US 2 Ernst Blofeldt Caracas, Venezuela ID ORDER_ID QTY DESCRIPTION PRICE 1 1 1 Aston Martin 120,000 2 1 1 Dinner Jacket 4,000 3 1 3 Champagne Veuve-Cliquot 200 4 2 100 Cat Food 1 5 2 1 Launch Pad 1,000,000 ORDER_ID TIMESTAMP STATUS 1 1985-04-30 09:48:00 ORDERED 2 1985-04-23 01:30:22 ORDERED 2 1985-04-25 08:30:00 SHIPPED 2 1985-05-14 21:37:00 DELIVERED { "first_name" : "James", "last_name" : "Bond", "address" : "Nassau, Bahamas, US", "items" : [ { ..., "description" : "Aston Martin", ... }, { ..., "description" : "Dinner Jacket", ... }, { ..., "description" : "Champagne...", ... } ], "tracking" : [ { ... "1985-04-30 09:48:00", ... "ORDERED" } ] }
  • 57. ORDERS TRACKING ITEMS ID FIRST_NAME LAST_NAME SHIPPING_ADDRESS 1 James Bond Nassau, Bahamas, US 2 Ernst Blofeldt Caracas, Venezuela ID ORDER_ID QTY DESCRIPTION PRICE 1 1 1 Aston Martin 120,000 2 1 1 Dinner Jacket 4,000 3 1 3 Champagne Veuve-Cliquot 200 4 2 100 Cat Food 1 5 2 1 Launch Pad 1,000,000 ORDER_ID TIMESTAMP STATUS 1 1985-04-30 09:48:00 ORDERED 2 1985-04-23 01:30:22 ORDERED 2 1985-04-25 08:30:00 SHIPPED 2 1985-05-14 21:37:00 DELIVERED { "first_name" : "James", "last_name" : "Bond", "address" : "Nassau, Bahamas, US", "items" : [ { ..., "description" : "Aston Martin", ... }, { ..., "description" : "Dinner Jacket", ... }, { ..., "description" : "Champagne...", ... } ], "tracking" : [ { ... "1985-04-30 09:48:00", ... "ORDERED" } ] }
  • 58. ORDERS TRACKING ITEMS ID FIRST_NAME LAST_NAME SHIPPING_ADDRESS 1 James Bond Nassau, Bahamas, US 2 Ernst Blofeldt Caracas, Venezuela ID ORDER_ID QTY DESCRIPTION PRICE 1 1 1 Aston Martin 120,000 2 1 1 Dinner Jacket 4,000 3 1 3 Champagne Veuve-Cliquot 200 4 2 100 Cat Food 1 5 2 1 Launch Pad 1,000,000 ORDER_ID TIMESTAMP STATUS 1 1985-04-30 09:48:00 ORDERED 2 1985-04-23 01:30:22 ORDERED 2 1985-04-25 08:30:00 SHIPPED 2 1985-05-14 21:37:00 DELIVERED
  • 59. ORDERS TRACKING ITEMS ID FIRST_NAME LAST_NAME SHIPPING_ADDRESS 1 James Bond Nassau, Bahamas, US 2 Ernst Blofeldt Caracas, Venezuela ID ORDER_ID QTY DESCRIPTION PRICE 1 1 1 Aston Martin 120,000 2 1 1 Dinner Jacket 4,000 3 1 3 Champagne Veuve-Cliquot 200 4 2 100 Cat Food 1 5 2 1 Launch Pad 1,000,000 ORDER_ID TIMESTAMP STATUS 1 1985-04-30 09:48:00 ORDERED 2 1985-04-23 01:30:22 ORDERED 2 1985-04-25 08:30:00 SHIPPED 2 1985-05-14 21:37:00 DELIVERED
  • 60. ORDERS TRACKING ITEMS ID FIRST_NAME LAST_NAME SHIPPING_ADDRESS 1 James Bond Nassau, Bahamas, US 2 Ernst Blofeldt Caracas, Venezuela ID ORDER_ID QTY DESCRIPTION PRICE 1 1 1 Aston Martin 120,000 2 1 1 Dinner Jacket 4,000 3 1 3 Champagne Veuve-Cliquot 200 4 2 100 Cat Food 1 5 2 1 Launch Pad 1,000,000 ORDER_ID TIMESTAMP STATUS 1 1985-04-30 09:48:00 ORDERED 2 1985-04-23 01:30:22 ORDERED 2 1985-04-25 08:30:00 SHIPPED 2 1985-05-14 21:37:00 DELIVERED { "first_name" : "Ernst", "last_name" : "Blofeldt", "address" : "Caracas, Venezuela" }
  • 61. ORDERS TRACKING ITEMS ID FIRST_NAME LAST_NAME SHIPPING_ADDRESS 1 James Bond Nassau, Bahamas, US 2 Ernst Blofeldt Caracas, Venezuela ID ORDER_ID QTY DESCRIPTION PRICE 1 1 1 Aston Martin 120,000 2 1 1 Dinner Jacket 4,000 3 1 3 Champagne Veuve-Cliquot 200 4 2 100 Cat Food 1 5 2 1 Launch Pad 1,000,000 ORDER_ID TIMESTAMP STATUS 1 1985-04-30 09:48:00 ORDERED 2 1985-04-23 01:30:22 ORDERED 2 1985-04-25 08:30:00 SHIPPED 2 1985-05-14 21:37:00 DELIVERED { "first_name" : "Ernst", "last_name" : "Blofeldt", "address" : "Caracas, Venezuela", "items" : [ { ..., "description" : "Cat Food", ... } ] }
  • 62. ORDERS TRACKING ITEMS ID FIRST_NAME LAST_NAME SHIPPING_ADDRESS 1 James Bond Nassau, Bahamas, US 2 Ernst Blofeldt Caracas, Venezuela ID ORDER_ID QTY DESCRIPTION PRICE 1 1 1 Aston Martin 120,000 2 1 1 Dinner Jacket 4,000 3 1 3 Champagne Veuve-Cliquot 200 4 2 100 Cat Food 1 5 2 1 Launch Pad 1,000,000 ORDER_ID TIMESTAMP STATUS 1 1985-04-30 09:48:00 ORDERED 2 1985-04-23 01:30:22 ORDERED 2 1985-04-25 08:30:00 SHIPPED 2 1985-05-14 21:37:00 DELIVERED { "first_name" : "Ernst", "last_name" : "Blofeldt", "address" : "Caracas, Venezuela", "items" : [ { ..., "description" : "Cat Food", ... }, { ..., "description" : "Launch Pad", ... } ] }
  • 63. ORDERS TRACKING ITEMS ID FIRST_NAME LAST_NAME SHIPPING_ADDRESS 1 James Bond Nassau, Bahamas, US 2 Ernst Blofeldt Caracas, Venezuela ID ORDER_ID QTY DESCRIPTION PRICE 1 1 1 Aston Martin 120,000 2 1 1 Dinner Jacket 4,000 3 1 3 Champagne Veuve-Cliquot 200 4 2 100 Cat Food 1 5 2 1 Launch Pad 1,000,000 ORDER_ID TIMESTAMP STATUS 1 1985-04-30 09:48:00 ORDERED 2 1985-04-23 01:30:22 ORDERED 2 1985-04-25 08:30:00 SHIPPED 2 1985-05-14 21:37:00 DELIVERED { "first_name" : "Ernst", "last_name" : "Blofeldt", "address" : "Caracas, Venezuela", "items" : [ { ..., "description" : "Cat Food", ... }, { ..., "description" : "Launch Pad", ... } ] }
  • 64. ORDERS TRACKING ITEMS ID FIRST_NAME LAST_NAME SHIPPING_ADDRESS 1 James Bond Nassau, Bahamas, US 2 Ernst Blofeldt Caracas, Venezuela ID ORDER_ID QTY DESCRIPTION PRICE 1 1 1 Aston Martin 120,000 2 1 1 Dinner Jacket 4,000 3 1 3 Champagne Veuve-Cliquot 200 4 2 100 Cat Food 1 5 2 1 Launch Pad 1,000,000 ORDER_ID TIMESTAMP STATUS 1 1985-04-30 09:48:00 ORDERED 2 1985-04-23 01:30:22 ORDERED 2 1985-04-25 08:30:00 SHIPPED 2 1985-05-14 21:37:00 DELIVERED { "first_name" : "Ernst", "last_name" : "Blofeldt", "address" : "Caracas, Venezuela", "items" : [ { ..., "description" : "Cat Food", ... }, { ..., "description" : "Launch Pad", ... } ], "tracking" : [ { ... "1985-04-23 01:30:22", ... "ORDERED" } ] }
  • 65. ORDERS TRACKING ITEMS ID FIRST_NAME LAST_NAME SHIPPING_ADDRESS 1 James Bond Nassau, Bahamas, US 2 Ernst Blofeldt Caracas, Venezuela ID ORDER_ID QTY DESCRIPTION PRICE 1 1 1 Aston Martin 120,000 2 1 1 Dinner Jacket 4,000 3 1 3 Champagne Veuve-Cliquot 200 4 2 100 Cat Food 1 5 2 1 Launch Pad 1,000,000 ORDER_ID TIMESTAMP STATUS 1 1985-04-30 09:48:00 ORDERED 2 1985-04-23 01:30:22 ORDERED 2 1985-04-25 08:30:00 SHIPPED 2 1985-05-14 21:37:00 DELIVERED { "first_name" : "Ernst", "last_name" : "Blofeldt", "address" : "Caracas, Venezuela", "items" : [ { ..., "description" : "Cat Food", ... }, { ..., "description" : "Launch Pad", ... } ], "tracking" : [ { ... "1985-04-23 01:30:22", ... "ORDERED" }, { ... "1985-04-25 08:30:00", ... "SHIPPED" } ] }
  • 66. ORDERS TRACKING ITEMS ID FIRST_NAME LAST_NAME SHIPPING_ADDRESS 1 James Bond Nassau, Bahamas, US 2 Ernst Blofeldt Caracas, Venezuela ID ORDER_ID QTY DESCRIPTION PRICE 1 1 1 Aston Martin 120,000 2 1 1 Dinner Jacket 4,000 3 1 3 Champagne Veuve-Cliquot 200 4 2 100 Cat Food 1 5 2 1 Launch Pad 1,000,000 ORDER_ID TIMESTAMP STATUS 1 1985-04-30 09:48:00 ORDERED 2 1985-04-23 01:30:22 ORDERED 2 1985-04-25 08:30:00 SHIPPED 2 1985-05-14 21:37:00 DELIVERED { "first_name" : "Ernst", "last_name" : "Blofeldt", "address" : "Caracas, Venezuela", "items" : [ { ..., "description" : "Cat Food", ... }, { ..., "description" : "Launch Pad", ... } ], "tracking" : [ { ... "1985-04-23 01:30:22", ... "ORDERED" }, { ... "1985-04-25 08:30:00", ... "SHIPPED" }, { ... "1985-05-14 21:37:00", .. "DELIVERED" } ] }
  • 67. ORDERS TRACKING ITEMS ID FIRST_NAME LAST_NAME SHIPPING_ADDRESS 1 James Bond Nassau, Bahamas, US 2 Ernst Blofeldt Caracas, Venezuela ID ORDER_ID QTY DESCRIPTION PRICE 1 1 1 Aston Martin 120,000 2 1 1 Dinner Jacket 4,000 3 1 3 Champagne Veuve-Cliquot 200 4 2 100 Cat Food 1 5 2 1 Launch Pad 1,000,000 ORDER_ID TIMESTAMP STATUS 1 1985-04-30 09:48:00 ORDERED 2 1985-04-23 01:30:22 ORDERED 2 1985-04-25 08:30:00 SHIPPED 2 1985-05-14 21:37:00 DELIVERED { "first_name" : "Ernst", "last_name" : "Blofeldt", "address" : "Caracas, Venezuela", "items" : [ { ..., "description" : "Cat Food", ... }, { ..., "description" : "Launch Pad", ... } ], "tracking" : [ { ... "1985-04-23 01:30:22", ... "ORDERED" }, { ... "1985-04-25 08:30:00", ... "SHIPPED" }, { ... "1985-05-14 21:37:00", .. "DELIVERED" } ] }
  • 68. ORDERS TRACKING ITEMS ID FIRST_NAME LAST_NAME SHIPPING_ADDRESS 1 James Bond Nassau, Bahamas, US 2 Ernst Blofeldt Caracas, Venezuela ID ORDER_ID QTY DESCRIPTION PRICE 1 1 1 Aston Martin 120,000 2 1 1 Dinner Jacket 4,000 3 1 3 Champagne Veuve-Cliquot 200 4 2 100 Cat Food 1 5 2 1 Launch Pad 1,000,000 ORDER_ID TIMESTAMP STATUS 1 1985-04-30 09:48:00 ORDERED 2 1985-04-23 01:30:22 ORDERED 2 1985-04-25 08:30:00 SHIPPED 2 1985-05-14 21:37:00 DELIVERED Done!
  • 70. #MDBlocal Fan-In and Fan-Out ETL Job Number of Database Operations per MongoDB Document 3/ n 1
  • 71. #MDBlocal • Yes. Although not as straightforward as you might think. Did you just explain to me what a JOIN is? • No. Co-Iteration works from multiple data sources. NAME ITEM TRACKING James Bond Aston Martin ORDERED James Bond Aston Martin SHIPPED James Bond Dinner Jacket ORDERED James Bond Dinner Jacket SHIPPED James Bond Champagne ORDERED James Bond Champagne SHIPPED
  • 72. #MDBlocal Oh, and one more thing…
  • 74. #MDBlocal Fan-In and Fan-out ETL Job Number of Database Operations per MongoDB Document 3/ n 1/1000
  • 76. #MDBlocal • Common Mistakes to Watch Out For • Nested Queries • Building Documents in the Database • Loading Everything into Memory • The Co-Iteration Pattern • Open All Tables at Once • Perform a Single Pass over Them • Build Documents as You Go Along • Don't Forget Batching and Threading Summary