Analytics and Machine Learning with Spark and MongoDB
Download as PPTX, PDF1 like2,354 views
The document discusses various aspects of utilizing Spark and MongoDB for machine learning, particularly focusing on clustering algorithms and data processing. It provides a series of configurations for Spark shell commands with MongoDB for accessing and managing precipitation and weather station data. Additionally, it references resources and examples related to Spark and MongoDB integrations.
![Interactive
Shell
Easy[ier] API
Caching](https://image.slidesharecdn.com/mongodb-spark-mdbw-70655b61-7748-4c26-b299-426da2bb9868-1278448928-180131221955/85/Analytics-and-Machine-Learning-with-Spark-and-MongoDB-9-320.jpg)
Analytics and Machine Learning with Spark and MongoDB
- 1. #MDBlocal Bryan Reinero, Product Manager Analytics and Machine Learning with Spark and MongoDB
- 24. { "_id" : "01006303201304010015QPCP", "value" : 0, "station" : "006303", "time" : "0015", "state" : "01", "year" : 2013, "month" : 4, "day" : 1 } Precipitation Data
- 25. { "_id" : NumberLong(10957), "NCDCSTN_ID" : "20000282", "NWSLI_ID" : "BOZA1", "GHCND_ID" : "USC00010957", "NAME_COOP_SHORT" : "BOAZ", "FIPS_COUNTRY_NAME" : "UNITED STATES", "STATE_PROV" : "AL", "COUNTY" : "MARSHALL", "NWS_CLIM_DIV" : "02", "NWS_CLIM_DIV_NAME" : "APPALACHIAN MOUNTAINS", "LON_DEC" : -86.1633, "LAT_DEC" : 34.2008, "LAT_LON_PRECISION" : "DDdddd", "ELEV_GROUND" : "1070", "ELEV_GROUND_UNIT" : "FEET", "UTC_OFFSET" : -6, "NWS_REGION" : "SOUTHERN", "NWS_WFO" : "HUN", "COOP_SOD" : "Y", "COOP_HPD" : "Y" } Weather Station Data
- 26. { "_id" : NumberLong(10957), "NCDCSTN_ID" : "20000282", "NWSLI_ID" : "BOZA1", "GHCND_ID" : "USC00010957", "NAME_COOP_SHORT" : "BOAZ", "FIPS_COUNTRY_NAME" : "UNITED STATES", "STATE_PROV" : "AL", "COUNTY" : "MARSHALL", "NWS_CLIM_DIV" : "02", "NWS_CLIM_DIV_NAME" : "APPALACHIAN MOUNTAINS", "LON_DEC" : -86.1633, "LAT_DEC" : 34.2008, "LAT_LON_PRECISION" : "DDdddd", "ELEV_GROUND" : "1070", "ELEV_GROUND_UNIT" : "FEET", "UTC_OFFSET" : -6, "NWS_REGION" : "SOUTHERN", "NWS_WFO" : "HUN", "COOP_SOD" : "Y", "COOP_HPD" : "Y" }
- 27. ./spark-2.2.0/bin/spark-shell --conf "spark.mongodb.input.uri=mongodb://10.11.12.13/kmeans.log" --conf "spark.mongodb.output.uri=mongodb://10.11.12.13/kmeans.cluster" --packages org.mongodb.spark:mongo-spark-connector_2.11:2.0.0
- 28. ./spark-2.1.0-bin-hadoop2.7/bin/spark-shell --conf "spark.mongodb.input.uri=mongodb://10.11.12.13/kmeans.log" --conf "spark.mongodb.output.uri=mongodb://10.11.12.13/kmeans.cluster" --packages org.mongodb.spark:mongo-spark-connector_2.11:2.0.0
- 29. ./spark-2.1.0-bin-hadoop2.7/bin/spark-shell --conf "spark.mongodb.input.uri=mongodb://10.0.0.10/kmeans.log" --conf "spark.mongodb.output.uri=mongodb://10.0.0.10/kmeans.cluster" --packages org.mongodb.spark:mongo-spark-connector_2.11:2.0.0
- 30. ./spark-2.1.0-bin-hadoop2.7/bin/spark-shell --conf "spark.mongodb.input.uri=mongodb://10.0.0.10/kmeans.log" --conf "spark.mongodb.output.uri=mongodb://10.0.0.10/kmeans.cluster" --packages org.mongodb.spark:mongo-spark-connector_2.11:2.0.0
- 31. Demo Time
- 33. Resource s MongoDB University Course M: 233 Spark Connector Examples in Repo