Using MongoDB with Hadoop & Spark
Download as PPTX, PDF1 like3,295 views
The document discusses the integration of MongoDB with Hadoop and Spark for data management, highlighting their capabilities in handling unstructured data, fault tolerance, and scalable analytics. It provides insights into use cases such as behavioral analytics and fraud detection, along with implementation details and code snippets for MapReduce tasks. Additionally, it emphasizes the benefits of combining these technologies, including machine learning access and reduced deployment complexity.
![15
EnronMailMapper Class
@Override
public void map(final Object key, final BSONObject val,
final Context context)
throws IOException, InterruptedException {
BSONObject headers = (BSONObject) val.get("headers");
String to = (String) headers.get("To");
if (null != to) {
String[] recipients = to.split(",");
for (final String recip1 : recipients) {
String recip = recip1.trim();
if (recip.length() > 0) {
mp.setFrom((String) key);
mp.setTo(recip);
context.write(mp, intw);
}
}
}
}](https://image.slidesharecdn.com/mongodbhadoopspark-151014194718-lva1-app6892/85/Using-MongoDB-with-Hadoop-Spark-15-320.jpg)
![18
MapReduce
map() {
String to = (String) headers.get("To");
if (null != to) {
String[] recipients = to.split(",");
for (final String recip1 : recipients) {
String recip = recip1.trim();
if (recip.length() > 0) {
mp.setFrom((String) key);
mp.setTo(recip);
context.write(mp, intw);
}
}
}](https://image.slidesharecdn.com/mongodbhadoopspark-151014194718-lva1-app6892/85/Using-MongoDB-with-Hadoop-Spark-18-320.jpg)
![19
MapReduce
map() {
String to = (String) headers.get("To");
if (null != to) {
String[] recipients = to.split(",");
for (final String recip1 : recipients) {
String recip = recip1.trim();
if (recip.length() > 0) {
mp.setFrom((String) key);
mp.setTo(recip);
context.write(mp, intw);
}
}
}](https://image.slidesharecdn.com/mongodbhadoopspark-151014194718-lva1-app6892/85/Using-MongoDB-with-Hadoop-Spark-19-320.jpg)
![20
MapReduce
map() {
String to = (String) headers.get("To");
if (null != to) {
String[] recipients = to.split(",");
for (final String recip1 : recipients) {
String recip = recip1.trim();
if (recip.length() > 0) {
mp.setFrom((String) key);
mp.setTo(recip);
context.write(mp, intw);
}
}
}](https://image.slidesharecdn.com/mongodbhadoopspark-151014194718-lva1-app6892/85/Using-MongoDB-with-Hadoop-Spark-20-320.jpg)
![21
MapReduce
{from: “Ken”,
to: “Jeff” }
map() {
String to = (String) headers.get("To");
if (null != to) {
String[] recipients = to.split(",");
for (final String recip1 : recipients) {
String recip = recip1.trim();
if (recip.length() > 0) {
mp.setFrom((String) key);
mp.setTo(recip);
context.write(mp, intw);
}
}
}](https://image.slidesharecdn.com/mongodbhadoopspark-151014194718-lva1-app6892/85/Using-MongoDB-with-Hadoop-Spark-21-320.jpg)
![22
MapReduce
{from: “Ken”,
to: “Jeff” }
map() {
String to = (String) headers.get("To");
if (null != to) {
String[] recipients = to.split(",");
for (final String recip1 : recipients) {
String recip = recip1.trim();
if (recip.length() > 0) {
mp.setFrom((String) key);
mp.setTo(recip);
context.write(mp, intw);
}
}
}
{from: “Ken”,
to: “Andrew” }
{from: “Ken”,
to: “Rebecca” }](https://image.slidesharecdn.com/mongodbhadoopspark-151014194718-lva1-app6892/85/Using-MongoDB-with-Hadoop-Spark-22-320.jpg)
Using MongoDB with Hadoop & Spark
- 1. MongoDB, Hadoop, & Spark { Name: ‘Bryan Reinero’, Title: ‘Developer Advocate’, Twitter: ‘@blimpyacht’, Email: ‘[email protected]’, Web: ‘mongodb.com/bryan’ }
- 3. 3 Data Management Hadoop Fault tolerance Batch processing Coarse-grained operations Unstructured Data MongoDB High availability Mutable data Fine-grained operations Flexible Schemas
- 4. 4 Data Management Hadoop Offline Processing Analytics Data Warehousing MongoDB Online Operations Application Operational
- 5. 5 Use Cases • Behavioral analytics • Segmentation • Fraud detection • Prediction • Pricing analytics • Sales analytics
- 6. 6 Typical Implementations Data ServicesHDFS Application Database
- 9. 9 Yet Another Resource Negotiator Compute Nodes Client NODE MANAGER NODE MANAGER RESOURCE MANAGER APPLICATION MASTER CONTAINER CONTAINER CONTAINER Bin/hadoop jar MyJob.jar MongoDB_Hadoop_Connector.jar
- 10. 10 Running an Example COMPUTE NODE Client NODE MANAGER NODE MANAGER RESOURCE MANAGER APPLICATION MASTER CONTAINER CONTAINER CONTAINER Bin/hadoop jar MyJob.jar MongoDB_Hadoop_Connector.jar
- 11. 11 Running an Example COMPUTE NODE Client NODE MANAGER NODE MANAGER RESOURCE MANAGER APPLICATION MASTER CONTAINER CONTAINER CONTAINER Bin/hadoop jar MyJob.jar MongoDB_Hadoop_Connector.jar
- 14. 14 What You’re Gonna Need A reducer class extends org.apache.hadoop.mapreduce.Reducer A mapper class extends org.apache.hadoop.mapreduce.Mapper
- 15. 15 EnronMailMapper Class @Override public void map(final Object key, final BSONObject val, final Context context) throws IOException, InterruptedException { BSONObject headers = (BSONObject) val.get("headers"); String to = (String) headers.get("To"); if (null != to) { String[] recipients = to.split(","); for (final String recip1 : recipients) { String recip = recip1.trim(); if (recip.length() > 0) { mp.setFrom((String) key); mp.setTo(recip); context.write(mp, intw); } } } }
- 16. 16 EnronMailReducer Class @Override public void reduce(final MailPair pKey, final Iterable<IntWritable> pValues, final Context pContext) throws IOException, InterruptedException { int sum = 0; for (final IntWritable value : pValues) { sum += value.get(); } BSONObject outDoc = BasicDBObjectBuilder.start() .add("f", pKey.getFrom()) .add("t", pKey.getTo()).get(); reduceResult.setDoc(outDoc); intw.set(sum); pContext.write(reduceResult, intw); }
- 17. 17 MapReduce
- 18. 18 MapReduce map() { String to = (String) headers.get("To"); if (null != to) { String[] recipients = to.split(","); for (final String recip1 : recipients) { String recip = recip1.trim(); if (recip.length() > 0) { mp.setFrom((String) key); mp.setTo(recip); context.write(mp, intw); } } }
- 19. 19 MapReduce map() { String to = (String) headers.get("To"); if (null != to) { String[] recipients = to.split(","); for (final String recip1 : recipients) { String recip = recip1.trim(); if (recip.length() > 0) { mp.setFrom((String) key); mp.setTo(recip); context.write(mp, intw); } } }
- 20. 20 MapReduce map() { String to = (String) headers.get("To"); if (null != to) { String[] recipients = to.split(","); for (final String recip1 : recipients) { String recip = recip1.trim(); if (recip.length() > 0) { mp.setFrom((String) key); mp.setTo(recip); context.write(mp, intw); } } }
- 21. 21 MapReduce {from: “Ken”, to: “Jeff” } map() { String to = (String) headers.get("To"); if (null != to) { String[] recipients = to.split(","); for (final String recip1 : recipients) { String recip = recip1.trim(); if (recip.length() > 0) { mp.setFrom((String) key); mp.setTo(recip); context.write(mp, intw); } } }
- 22. 22 MapReduce {from: “Ken”, to: “Jeff” } map() { String to = (String) headers.get("To"); if (null != to) { String[] recipients = to.split(","); for (final String recip1 : recipients) { String recip = recip1.trim(); if (recip.length() > 0) { mp.setFrom((String) key); mp.setTo(recip); context.write(mp, intw); } } } {from: “Ken”, to: “Andrew” } {from: “Ken”, to: “Rebecca” }
- 23. 23 MapReduce
- 24. 24 MapReduce from: “Ken”, to: “Jeff”
- 25. 25 MapReduce from: “Ken”, to: “Jeff”
- 26. 26 MapReduce from: “Ken”, to: “Jeff”
- 27. 27 MapReduce from: “Ken”, to: “Andrew”
- 28. 28 MapReduce from: “Ken”, to: “Andrew”
- 29. 29 MapReduce from: “Ken”, to: “Andrew”
- 30. 30 MapReduce from: “Ken”, to: “Andrew”
- 31. 31 MapReduce public void reduce() { int sum = 0; for (final IntWritable value:pValues) sum += value.get(); BSONObject outDoc = BasicDBObjectBuilder.start() .add("f", pKey.getFrom()) .add("t", pKey.getTo()).get(); reduceResult.setDoc(outDoc); intw.set(sum); pContext.write(reduceResult, intw); }
- 32. 32 MapReduce public void reduce() { int sum = 0; for (final IntWritable value:pValues) sum += value.get(); BSONObject outDoc = BasicDBObjectBuilder.start() .add("f", pKey.getFrom()) .add("t", pKey.getTo()).get(); reduceResult.setDoc(outDoc); intw.set(sum); pContext.write(reduceResult, intw); }
- 33. Sharding & Input Splits
- 34. 34 Sharding Complete Data Set Define shard key 5906 84531002 69983880
- 35. 35 Sharding Chunk Chunk Define shard key on title 5906 84531002 69983880
- 36. 36 Sharding Chunk Chunk ChunkChunk Define shard key on title 5906 84531002 69983880
- 37. 37 Shard 1 Shard 2 Shard 3 Shard 4 Load Balancing
- 38. 38
- 40. 40 extends MongoSplitter class List<InputSplit> calculateSplits()
- 43. Spark Integration JavaPairRDD<Object, BSONObject> documents = sc.newAPIHadoopRDD( mongodbConfig, MongoInputFormat.class, Object.class, BSONObject.class );
- 44. Spark Integrations to Come…. New Spark Connector • Filter source data with Aggregation Framework • Spark SQL • Dataframes
- 47. A Primary B Secondary C Secondary Application Servers Indexes for CRUD Indexes for Analytics ReadPreference( “Reporting” )
- 48. MongoDB + Hadoop + Spark Benefits • Access to machine learning libs • Closed loop processing • Reduced deployment complexity • Leverage MongoDB caching • Persist intermediary results
- 49. Thanks! { name: ‘Bryan Reinero’, title: ‘Developer Advocate’, twitter: ‘@blimpyacht’, code: ‘github.com/breinero’ email: ‘[email protected]’ }