Live Demo: Introducing the Spark Connector for MongoDB
Download as PPTX, PDF2 likes1,711 views
The document outlines a presentation on integrating MongoDB with Apache Spark, focusing on distributed computing and data processing via Spark. It discusses the MongoDB Spark connector, configuration settings for data input and output, and various transformations and actions applicable to RDD (Resilient Distributed Datasets). Additionally, it provides links to resources for further learning on MongoDB and Spark integration.
Live Demo: Introducing the Spark Connector for MongoDB
- 1. MongoDBEurope2016 Old Billingsgate, London 15th November Distributed Ledgers, Blockchain + MongoDB Bryan Reinero
- 2. MongoDB Connector For Spark
- 15. Transformations filter( func ) union( func ) intersection( set ) distinct( n ) map( function )
- 18. Actions collect() count() first() take( n ) reduce( function )
- 30. { "_id" : ObjectId("578be1fe1fe699f2deb80807"), "user_id" : 196, "movie_id" : 242, "rating" : 3, "timestamp" : 881250949 }
- 31. ./bin/spark-shell --conf "spark.mongodb.input.uri=mongodb://127.0.0.1/movies.movie_ratings" --conf "spark.mongodb.output.uri=mongodb://127.0.0.1/movies.user_recommendations" --packages org.mongodb.spark:mongo-spark-connector_2.10:1.0.0
- 32. ./bin/spark-shell --conf "spark.mongodb.input.uri=mongodb://127.0.0.1/movies.movie_ratings" --conf "spark.mongodb.output.uri=mongodb://127.0.0.1/movies.user_recommendations" --packages org.mongodb.spark:mongo-spark-connector_2.10:1.0.0
- 33. ./bin/spark-shell --conf "spark.mongodb.input.uri=mongodb://127.0.0.1/movies.movie_ratings" --conf "spark.mongodb.output.uri=mongodb://127.0.0.1/movies.user_recommendations" --packages org.mongodb.spark:mongo-spark-connector_2.10:1.0.0
- 34. ./bin/spark-shell --conf "spark.mongodb.input.uri=mongodb://127.0.0.1/movies.movie_ratings" --conf "spark.mongodb.output.uri=mongodb://127.0.0.1/movies.user_recommendations" --packages org.mongodb.spark:mongo-spark-connector_2.10:1.0.0
- 35. import com.mongodb.spark._ import com.mongodb.spark.rdd.MongoRDD import org.bson.Document val rdd = sc.loadFromMongoDB() for( doc <- rdd.take( 10 ) ) println( doc )
- 36. Read Config Write Config
- 37. Aggregation Filters $match | $project | $group
- 40. val aggRdd = rdd.withPipeline( Seq( Document.parse( "{ $match: { Country: "USA" } }" ) ) )
- 42. RDD + Schema = Dataframe