Mongo db &_spark
Download as PPTX, PDF5 likes1,364 views
The document describes Spark and how it relates to other big data technologies like HDFS, YARN, and Mesos. It shows how Spark can be used both with these technologies or on its own. Key Spark concepts discussed include Resilient Distributed Datasets (RDDs), transformations, actions, and lineage. It also provides examples of using Spark with MongoDB to read data from a MongoDB collection into an RDD.
Mongo db &_spark
- 1. & Spark
- 2. & Spark
- 28. execut or Worker Node execut or Worker Node Driver Resilient Distributed Datasets
- 29. Resilient Distributed Datasets f(x’’) = yParellelize = xt(x) = x’t(x’) = x’’
- 30. t(x) = x’t(x’) = x’’f(x’’) = x’’’Parellelize = x Parallelization
- 31. t(x) = x’t(x’) = x’’f(x’’) = x’’’Parellelize = x Transformations
- 32. Tranformations filter( func ) union( func ) intersection( set ) distinct( n ) map( function )
- 33. t(x) = x’t(x’) = x’’f(x’’) = x’’’Parellelize = x Action
- 34. Actions collect() count() first() take( n ) reduce( function )
- 35. f(x) = x’f(x’) = x’’t(x’’) = x’’’Parellelize = x Lineage
- 36. Lineage
- 37. Lineage
- 38. Lineage
- 39. Lineage
- 41. { "_id" : ObjectId("4f16fc97d1e2d32371003e27"), "body" : "the scrimmage is still up in the air. "subFolder" : "notes_inbox", "mailbox" : "bass-e", "filename" : "450.", "headers" : { "X-cc" : "", "From" : "[email protected]", "Subject" : "Re: Plays and other information", "X-Folder" : "Eric_Bass_Dec2000Notes FoldersNotes inbox", "Content-Transfer-Encoding" : "7bit", "X-bcc" : "", "To" : "[email protected]", "X-Origin" : "Bass-E", "X-FileName" : "ebass.nsf", "X-From" : "Michael Simmons", "Date" : "Tue, 14 Nov 2000 08:22:00 -0800 (PST)", "X-To" : "Eric Bass", "Message-ID" : "<6884142.1075854677416.JavaMail.evans@thyme>", "Content-Type" : "text/plain; charset=us-ascii", "Mime-Version" : "1.0" } }
- 42. { "_id" : "[email protected]|[email protected]", "value" : 2 } { "_id" : "[email protected]|[email protected]", "value" : 2 } { "_id" : "[email protected]|[email protected]", "value" : 2 }
- 44. Spark Configuration Configuration conf = new Configuration(); conf.set( "mongo.job.input.format", "com.mongodb.hadoop.MongoInputFormat” ); conf.set( "mongo.input.uri", "mongodb://localhost:27017/db.collection” );
- 45. Spark Context JavaPairRDD<Object, BSONObject> documents = context.newAPIHadoopRDD( conf, MongoInputFormat.class, Object.class, BSONObject.class );
- 46. Spark Context JavaPairRDD<Object, BSONObject> documents = context.newAPIHadoopRDD( conf, MongoInputFormat.class, Object.class, BSONObject.class );
- 47. Spark Context JavaPairRDD<Object, BSONObject> documents = context.newAPIHadoopRDD( conf, MongoInputFormat.class, Object.class, BSONObject.class );
- 48. Spark Context JavaPairRDD<Object, BSONObject> documents = context.newAPIHadoopRDD( conf, MongoInputFormat.class, Object.class, BSONObject.class );
- 49. Spark Context JavaPairRDD<Object, BSONObject> documents = context.newAPIHadoopRDD( conf, MongoInputFormat.class, Object.class, BSONObject.class );
- 51. Deployment Artifacts Hadoop Connector Jar Fat Jar Java Driver Jar
- 52. Spark Submit /usr/local/spark-1.5.1/bin/spark-submit --class com.mongodb.spark.examples.DataframeExample --master local Examples-1.0-SNAPSHOT.jar
- 54. JavaRDD<Message> messages = documents.map ( new Function<Tuple2<Object, BSONObject>, Message>() { public Message call(Tuple2<Object, BSONObject> tuple) { BSONObject header = (BSONObject)tuple._2.get("headers"); Message m = new Message(); m.setTo( (String) header.get("To") ); m.setX_From( (String) header.get("From") ); m.setMessage_ID( (String) header.get( "Message-ID" ) ); m.setBody( (String) tuple._2.get( "body" ) ); return m; } } );
- 55. & Spack DEMO
- 62. THANK S!{ Name: ‘Bryan Reinero’, Title: ‘Developer Advocate’, Twitter: ‘@blimpyacht’, Email: ‘[email protected]’ }
Editor's Notes
- #29: A fault-tolerant collection of elements operated on in parallel best suited for batch applications
- #45: MongoInputFormat allows us to read from a live MongoDB instance. We could also use BSONFileInputFormat to read BSON snapshots.
- #46: JavaPamongodbConfig
- #47: JavaPamongodbConfig
- #48: JavaPamongodbConfig
- #49: JavaPamongodbConfig
- #50: JavaPamongodbConfig