Mongo db and hadoop driving business insights - final
Download as PPTX, PDF6 likes1,352 views
The document discusses the complementary roles of MongoDB and Hadoop in data management, highlighting their respective use cases and advantages. MongoDB is noted for its real-time analytics and ease of use, while Hadoop excels in processing large datasets for batch analytics. The document also outlines the integration between MongoDB and Hadoop through connectors, enabling efficient data movement and processing capabilities.
Mongo db and hadoop driving business insights - final
- 2. MongoDB and Hadoop Luke Lovett Software Engineer, MongoDB
- 3. Agenda • Complementary Approaches to Data • MongoDB & Hadoop Use Cases • MongoDB Connector Overview and Features • Demo
- 4. Complementary Approaches to Data
- 5. Operational: MongoDB Real-Time Analytics Product/Asset Catalogs Security & Fraud Internet of Things Mobile Apps Customer Data Mgmt Single View Social Churn Analysis Recommender Warehouse & ETL Risk Modeling Trade Surveillance Predictive Analytics Ad Targeting Sentiment Analysis
- 6. MongoDB • Store and read data frequently • Easy administration • Built-in analytical tools – aggregation framework – JavaScript MapReduce – Geo/text indexes
- 7. Analytical: Hadoop Real-Time Analytics Product/Asset Catalogs Security & Fraud Internet of Things Mobile Apps Customer Data Mgmt Single View Social Churn Analysis Recommender Warehouse & ETL Risk Modeling Trade Surveillance Predictive Analytics Ad Targeting Sentiment Analysis
- 8. Hadoop The Apache Hadoop software library is a framework that allows for the distributed processing of large data sets across clusters of computers using simple programming models. • Terabyte and Petabyte datasets • Data warehousing • Advanced analytics
- 9. Operational vs. Analytical: Lifecycle Real-Time Analytics Product/Asset Catalogs Security & Fraud Internet of Things Mobile Apps Customer Data Mgmt Single View Social Churn Analysis Recommender Warehouse & ETL Risk Modeling Trade Surveillance Predictive Analytics Ad Targeting Sentiment Analysis
- 10. MongoDB & Hadoop Use Cases
- 11. Batch Aggregation Applicatio ns powered by Analysis powered by MongoDB Connector for Hadoop ● Need more than MongoDB aggregation ● Need offline processing ● Results sent back to MongoDB ● Can be left as BSON on HDFS for further analysis
- 12. Commerce Applicatio ns powered by Analysis powered by • Products & Inventory • Recommended products • Customer profile • Session management • Elastic pricing • Recommendation models • Predictive analytics • Clickstream history MongoDB Connector for Hadoop
- 13. Fraud Detection Payments Nightly Analysis Fraud modeling MongoDB Connector for Hadoop Results Cache Online payments processing 3rd Party Data Sources Fraud Detection query only query only
- 14. MongoDB Connector for Hadoop
- 15. Connector Overview Hadoop Map Reduce, Hive, Pig, Spark HDFS / S3 Hadoop Connector Text Files Hadoop Connector BSON Files MongoDB Single Node, Replica Set, Cluster Apache Hadoop / Cloudera CDH / Hortonworks HDP / Amazon EMR
- 16. Data Movement Dynamic queries to MongoDB vs. BSON snapshots in HDFS Dynamic queries with most recent data Puts load on operational database Snapshots move load to Hadoop Snapshots add predictable load to MongoDB
- 17. Connector Operation 1. Split according to given InputFormat - many options available for reading from live cluster - configure key pattern, split strategy 1. Write splits file 2. Output to BSON file or live MongoDB - BSON file splits written automatically for future tasks - Mongo insertion round-robin across collections
- 18. Getting Splits • Split on a sharded cluster – Split by chunk – Split by shard • Splits on replica set/standalone – splitVector command • BSON files – specify max docs – split per input file MongoDB Connector for Hadoop Config Servers Shard Chunk Chunk Chunk Mongos Shard Chunk Chunk Chunk Shard Chunk Chunk Chunk
- 19. MongoDB Connector for Hadoop Config Servers Getting Splits • Split on a sharded cluster – Split by chunk – Split by shard • Splits on replica set/standalone – splitVector command • BSON files – specify max docs – split per input file Shard Chunk Chunk Chunk Mongos Shard Chunk Chunk Chunk Shard Chunk Chunk Chunk
- 20. MapReduce Configuration • MongoDB input – mongo.job.input.format = com.hadoop.MongoInputFormat – mongo.input.uri = mongodb://mydb:27017/db1.collection1 • MongoDB output – mongo.job.output.format = com.hadoop.MongoOutputFormat – mongo.output.uri = mongodb://mydb:27017/db1.collection2
- 21. MapReduce Configuration • BSON input/output – mongo.job.input.format = com.hadoop.BSONFileInputFormat – mapred.input.dir = hdfs:///tmp/database.bson – mongo.job.output.format = com.hadoop.BSONFileOutputFormat – mapred.output.dir = hdfs:///tmp/output.bson
- 22. Spark Usage • Use with MapReduce input/output formats • Create Configuration objects with input/output formats and data URI • Load/save data using SparkContext Hadoop file API
- 23. Hive Support CREATE TABLE mongo_users (id int, name string, age int) STORED BY "com.mongodb.hadoop.hive.MongoStorageHandler" WITH SERDEPROPERTIES("mongo.columns.mapping” = "_id,name,age”) TBLPROPERTIES("mongo.uri" = "mongodb://host:27017/test.users”) • Access collections as Hive tables • Use with MongoStorageHandler or BSONSerDe
- 24. Hive Support ● Types given by schema ● May use structs to project fields out of documents and ease access ● Can explode nested fields to make them top-level: {“customer”: {“name”: “Bart”}} can be accessed with “customer.name”. MongoDB Hive Primitive type (int, String, etc.) Primitive type (int, float, etc.) Document Row Sub-document Struct, Map, or exploded field Array Array or exploded field
- 25. Pig Mappings • Input: BSONLoader and MongoLoader data = LOAD ‘mongodb://mydb:27017/db.collection’ using com.mongodb.hadoop.pig.MongoLoader • Output: BSONStorage and MongoInsertStorage STORE records INTO ‘hdfs:///output.bson’ using com.mongodb.hadoop.pig.BSONStorage
- 26. Pig Mappings ● Organize and prune documents by specifying a schema ● Access full document in a Map without needing a schema MongoDB Pig Primitive type (int, String, etc.) Primitive type (int, chararray, etc.) Document Tuple (schema given) Document Tuple containing a Map (no schema) Sub-document Map Array Bag
- 27. Demo!
- 28. Questions?