MongoDB: Intro & Application for Big Data
19 likes4,905 views
The document discusses NoSQL databases and MongoDB in particular. It provides examples of MongoDB documents with different data types like objects, arrays, etc. It also shows how data is stored across multiple shard servers and accessed via mongos query routers. Configuration servers store sharded cluster metadata. Log data can be stored in MongoDB but only aggregated results data may be needed. HDFS is better for temporary raw log storage.
![{
"_id" : ObjectId("4dcd3ebc9278000000005158"),
"timestamp" : ISODate("2011-05-13T14:22:46.777Z"),
"binary" : BinData(0,""),
"string" : "abc",
"number" : 3,
"subobj" : {"subA": 1, "subB": { "subsubC": 2 }},
"array" : [1, 2, 3],
"dbref" : [_id1, _id2, _id3]
}](https://image.slidesharecdn.com/mongorakuten-111118210309-phpapp02/85/MongoDB-Intro-Application-for-Big-Data-9-320.jpg)
![{
"_id" : ObjectId("4dcd3ebc9278000000005158"),
"timestamp" : ISODate("2011-05-13T14:22:46.777Z"),
"binary" : BinData(0,""),
"string" : "abc",
"number" : 3,
"subobj" : {"subA": 1, "subB": 2 },
"array" : [1, 2, 3],
padding
}](https://image.slidesharecdn.com/mongorakuten-111118210309-phpapp02/85/MongoDB-Intro-Application-for-Big-Data-11-320.jpg)
![{
"_id" : ObjectId("4dcd3ebc9278000000005158"),
"timestamp" : ISODate("2011-05-13T14:22:46.777Z"),
"binary" : BinData(0,""),
"string" : "def",
"number" : 4,
"subobj" : {"subA": 1, "subB": 2 },
"array" : [1, 2, 3, 4, 5, 6],
"newkey" : "In-place"
}](https://image.slidesharecdn.com/mongorakuten-111118210309-phpapp02/85/MongoDB-Intro-Application-for-Big-Data-12-320.jpg)
![{
"_id" : ObjectId("4dcd3ebc9278000000005158"),
"timestamp" : ISODate("2011-05-13T14:22:46.777Z"),
"binary" : BinData(0,""),
"string" : "abc",
"number" : 3,
"subobj" : {"subA": 1, "subB": 2 },
"array" : [1, 2, 3],
}](https://image.slidesharecdn.com/mongorakuten-111118210309-phpapp02/85/MongoDB-Intro-Application-for-Big-Data-16-320.jpg)
MongoDB: Intro & Application for Big Data
- 9. { "_id" : ObjectId("4dcd3ebc9278000000005158"), "timestamp" : ISODate("2011-05-13T14:22:46.777Z"), "binary" : BinData(0,""), "string" : "abc", "number" : 3, "subobj" : {"subA": 1, "subB": { "subsubC": 2 }}, "array" : [1, 2, 3], "dbref" : [_id1, _id2, _id3] }
- 10. { "_id" : ObjectId("4dcd3ebc9278000000005158"), "nickname" : "doryokujin" },{ "_id" : ObjectId("4dcd3ebc9278000000005159"), "firstname" : "Takahiro", "lastname" : "Inoue", "mail" : "[email protected]", "twitter" : "@doryokujin" },...
- 11. { "_id" : ObjectId("4dcd3ebc9278000000005158"), "timestamp" : ISODate("2011-05-13T14:22:46.777Z"), "binary" : BinData(0,""), "string" : "abc", "number" : 3, "subobj" : {"subA": 1, "subB": 2 }, "array" : [1, 2, 3], padding }
- 12. { "_id" : ObjectId("4dcd3ebc9278000000005158"), "timestamp" : ISODate("2011-05-13T14:22:46.777Z"), "binary" : BinData(0,""), "string" : "def", "number" : 4, "subobj" : {"subA": 1, "subB": 2 }, "array" : [1, 2, 3, 4, 5, 6], "newkey" : "In-place" }
- 16. { "_id" : ObjectId("4dcd3ebc9278000000005158"), "timestamp" : ISODate("2011-05-13T14:22:46.777Z"), "binary" : BinData(0,""), "string" : "abc", "number" : 3, "subobj" : {"subA": 1, "subB": 2 }, "array" : [1, 2, 3], }
- 26. Cluster Shard Servers (Data) config Servers (Shard Configration) shard1 shard2 shard3 [ a, f ) [ k, n) [ o, t ) Chunk [ f, k ) [ n, o ) [ t, } ) mongos Servers (Routers)
- 27. Shard 1 ( mongos primary primary ) Shard cinfig Shard mongos
- 35. Every Server: Large HDD(500GB ) Large Memory(16GB ) Slave Delay in Master Data on Preparation for AmazonS3 User Error Master Data on S3 Non Sharding, Replica Set
- 38. From Text Logs: (Large) From mySQL: (Small) From Other NoSQL: (Middle) Temporary Raw data storage is HDFS, not MongoDB We only need result data to discover new features
- 39. Reduction First Second Aggregation Aggregation
- 40. ScientificPython
- 41. Fluent Structured logging Pluggable architecture Reliable forwarding e Event Collector Service
- 42. Fluent Structured logging Pluggable architecture Reliable forwarding e Event Collector Service Sadayuki Furuhashi Treasure Data, Inc. @frsyuki
- 43. “2011-04-01 host1 myapp: cmessage size=12MB user=me” 2011-04-01 myapp.message { “on_host”: ”host1”, 2011-04-01 myapp.message { ”combined”: true, “on_host”: ”host1”, ”combined”: true, “size”: 12000000, “size”: 12000000, “user”: “me” “user”: “me” } }
- 46. log log log log
- 48. aggregate aggregate aggregate aggregate log log log log key1 key2 key3 shuffle aggregate aggregate aggregate aggregate