How to leverage MongoDB for Big Data Analysis and Operations with MongoDB's Aggregation Framework and Map Reduce

Technical Support Engineer, 10gen
Gianfranco Palumbo
#bigdatajaspersoft
How to leverage MongoDB for
Big Data Analysis and
Operations
@MongoDBDublin

Join us this evening at Dublin
MUG
meetup.com/DublinMUG/

http://www.worldwidewebsize.com/
Exponential Data Growth

MongoDB solves our needs
• Ideal operational database
• Provides high performance for storage and
retrieval at large scale
• Has a robust query interface permitting intelligent
operations
• Is not a data processing engine, but provides
processing functionality

Data Processing in MongoDB
• Process in MongoDB using Map/Reduce
• Process in MongoDB using Aggregation
Framework
• Process outside MongoDB using Hadoop and
other external tools

The goal
Real Time
Analytics Engine
Data
SourceData
SourceData
Source

Solution goals
• Lots of data sources
• Lots of data from each source
High write volume
• Users can drill down into dataDynamic queries
• Lots of clients
• High request rate
Fast queries
• How long before an event appears
in a report?
Minimize delay
between collection &
query

Systems Architecture
Data
Sources
Asynchronous writes
Upserts avoid
unnecessary reads
Writes buffered in
RAM and flushed to
disk in bulk
Data
Sources
Data
Sources
Data
Sources
Spread writes over
multiple shards

Simple log storage
Design Pattern

Sample data
Original
Event Data
127.0.0.1 - frank [10/Jun/2013:13:55:36 -0700] "GET
/apache_pb.gif HTTP/1.0" 200 2326
“http://www.example.com/start.html" Mozilla/5.0 (Macintosh; U; Intel
Mac OS X 10_7_4; en-US)”
As JSON doc = {
_id: ObjectId('4f442120eb03305789000000'),
host: "127.0.0.1",
time: ISODate("2013-06-10T20:55:36Z"),
path: “/apache_pb.gif",
referer: “http://www.example.com/start.html",
user_agent: "Mozilla/5.0 (Macintosh; U; Intel Mac OS X
10_7_4; en-US)”
}
Insert to
MongoDB
db.logs.insert( doc )

Dynamic Queries
Find all logs for a
URL
db.logs.find( { ‘path’ : ‘/index.html’ } )
Find all logs for a
time range
db.logs.find( {
‘time’ : {
‘$gte’: new Date(2013, 0),
‘$lt’: new Date(2013, s1) }
} )
Find all logs for a
host over a range of
dates
db.logs.find( {
‘host’ : ‘127.0.0.1’,
‘time’ : {
‘$gte’: new Date(2013, 0),
‘$lt’: new Date(2013, 1) }
} )

Aggregation Framework
Requests
per day by
URL
db.logs.aggregate( [
{ '$match': {
'time': {
'$gte': new Date(2013, 0),
'$lt': new Date(2013, 1) } } },
{ '$project': {
'path': 1,
'date': {
'y': { '$year': '$time' },
'm': { '$month': '$time' },
'd': { '$dayOfMonth': '$time' } } } },
{ '$group': {
'_id': {
'p': '$path',
'y': '$date.y',
'm': '$date.m',
'd': '$date.d' },
'hits': { '$sum': 1 } } },
])

{
‘ok’: 1,
‘result’: [
{ '_id': {'p':’/index.html’,'y': 2013,'m': 1,'d': 1 },
'hits’: 124 },
'hits’: 245 },
'hits’: 322 },
'hits’: 175 },
'hits’: 94 }
]
}

Benefits
• Real-time
• Simple yet powerful interface
• Declared in JSON, executes in C++
• Runs inside MongoDB on local data 
• Adds load to your DB
• Limited how much data it can return

Roll-ups with map-
reduce
Design Pattern

Map Reduce – Map Phase
Generate hourly
rollups from log
data
var map = function() {
var key = {
p: this.path,
d: new Date(
this.ts.getFullYear(),
this.ts.getMonth(),
this.ts.getDate(),
this.ts.getHours(),
0, 0, 0) };
emit( key, { hits: 1 } );
}

Map Reduce – Reduce Phase
Generate hourly
rollups from log
data
var reduce = function(key, values) {
var r = { hits: 0 };
values.forEach(function(v) {
r.hits += v.hits;
});
return r;
}
)

MongoDB Map/Reduce
• Real-time
• Output directly to document or collection
• Runs inside MongoDB on local data
• V8 engine
• Adds load to your DB
• In JavaScript

APACHE HADOOP
Log
Aggregation
with MongoDB
as sink
More complex
aggregations or
integration with tools like
Mahout

MongoDB and Hadoop
• Away from data store
• Can leverage existing data processing infrastructure
• Can horizontally scale your data processing
• Offline batch processing
• Requires synchronization between store &
processor
• Infrastructure is much more complex

The Future of Big Data and
MongoDB

What is Big?
Big today is normal
tomorrow

http://www.worldwidewebsize.com/
Big is only getting bigger

IBM - http://www-01.ibm.com/software/data/bigdata/
90% of the data in the
world today has been
created in the last two
years

MongoDB enables
you to scale to the
redefinition of BIG

MongoDB is evolving
to enable you to process
the new BIG

Gianfranco Palumbo – slides tweeted from @MongoDBDublin
MongoDB is committed to
working with the best data
processing tools
• Map Reduce
• Aggregation Framework
• Hadoop adapter
– docs.mongodb.org/ecosystem/tutorial/getting-started-with-hadoop/
• Storm
– github.com/christkv/mongo-storm
• Disco
– github.com/mongodb/mongo-disco
• Spark (coming soon)

Technical Support Engineer, 10gen
Gianfranco Palumbo
#bigdatajaspersoft
Thank you
@MongoDBDublin

How to leverage MongoDB for Big Data Analysis and Operations with MongoDB's Aggregation Framework and Map Reduce

More Related Content

What's hot (20)

Viewers also liked (20)

Similar to How to leverage MongoDB for Big Data Analysis and Operations with MongoDB's Aggregation Framework and Map Reduce (20)

Recently uploaded (20)

How to leverage MongoDB for Big Data Analysis and Operations with MongoDB's Aggregation Framework and Map Reduce

Editor's Notes