MongoDB and Spark
Download as PPTX, PDF1 like1,680 views
The document outlines the integration of MongoDB with Apache Spark, detailing the key components of the Spark stack and how they interact with MongoDB. It discusses several use cases, including data processing and analytics, and provides links to relevant resources and demos. An emphasis is placed on the evolving capabilities and potential applications of this powerful technology pairing.
![Interactive Shell
Easy[ier] API
Caching](https://image.slidesharecdn.com/mongodbspark-dallas-190214145805/85/MongoDB-and-Spark-8-320.jpg)
![wget https://www-eu.apache.org/dist/spark/spark-2.4.0
/spark-2.4.0-bin-hadoop2.7.tgz
tar -xzvf spark-2.4.0-bin-hadoop2.7.tgz
spark-2.4.0-bin-hadoop2.7/bin/pyspark
Python 2.7.10 (default, Aug 17 2018, 17:41:52)
[GCC 4.2.1 Compatible Apple LLVM 10.0.0 (clang-1000.0.42)] on darwin
...
Using Python version 2.7.10 (default, Aug 17 2018 17:41:52)
SparkSession available as 'spark'.
>>>
Getting started with Spark - pyspark](https://image.slidesharecdn.com/mongodbspark-dallas-190214145805/85/MongoDB-and-Spark-12-320.jpg)
![48
Spark Streaming
Twitter
Feed
{
"statuses": [
{
"coordinates": null,
"favorited": false,
"truncated": false,
"created_at": "Mon Sep 24 03:35:21 +0000
2012",
"id_str": "250075927172759552",
"entities": {
"urls": [
],
"hashtags": [
{
"text": "freebandnames",
"indices": [
20,
34
]
}
],
"user_mentions": []
}
}
}](https://image.slidesharecdn.com/mongodbspark-dallas-190214145805/85/MongoDB-and-Spark-48-320.jpg)
![49
Spark Streaming
Spark
{
"statuses": [
{
"coordinates": null,
"favorited": false,
"truncated": false,
"created_at": "Mon Sep 24 03:35:21 +0000
2012",
"id_str": "250075927172759552",
"entities": {
"urls": [
],
"hashtags": [
{
"text": "freebandnames",
"indices": [
20,
34
]
}
],
"user_mentions": []
}
}
}
{
"time": "Mon Sep 24 03:35",
"freebandnames": 1
}
{
"statuses": [
{
"coordinates": null,
"favorited": false,
"truncated": false,
"created_at": "Mon Sep 24 03:35:21 +0000
2012",
"id_str": "250075927172759552",
"entities": {
"urls": [
],
"hashtags": [
{
"text": "freebandnames",
"indices": [
20,
34
]
}
],
"user_mentions": []
}
}
}
{
"statuses": [
{
"coordinates": null,
"favorited": false,
"truncated": false,
"created_at": "Mon Sep 24 03:35:21 +0000
2012",
"id_str": "250075927172759552",
"entities": {
"urls": [
],
"hashtags": [
{
"text": "freebandnames",
"indices": [
20,
34
]
}
],
"user_mentions": []
}
}
}
{
"statuses": [
{
"coordinates": null,
"favorited": false,
"truncated": false,
"created_at": "Mon Sep 24 03:35:21 +0000
2012",
"id_str": "250075927172759552",
"entities": {
"urls": [
],
"hashtags": [
{
"text": "freebandnames",
"indices": [
20,
34
]
}
],
"user_mentions": []
}
}
}
{
"time": "Mon Sep 24 03:35",
"freebandnames": 4
}](https://image.slidesharecdn.com/mongodbspark-dallas-190214145805/85/MongoDB-and-Spark-49-320.jpg)
MongoDB and Spark
- 1. MongoDB and Spark Getting MongoDB and Spark to play nice together
- 3. Agenda What we will unravel today!
- 4. 4 Agenda What Is Spark Overview Spark Stack Spark + MongoDB How to set up MongoDB and Spark? Integration Use Cases / Demo Datascience, Analytics Others
- 5. 5 Howdy! Who's this guy? Norberto Leite Lead Engineer @nleite MongoDB
- 9. 9 Delivering User Relevancy • Integrate data from many sources • Fast-cycle analytics • Real-time • Reliable
- 10. 10 Wearable Devices Embedded Systems Internet of Things Embedded Medical Devices
- 11. 11 Access complete patient history Avoid of conflicting prescriptions Clinical trials
- 12. wget https://www-eu.apache.org/dist/spark/spark-2.4.0 /spark-2.4.0-bin-hadoop2.7.tgz tar -xzvf spark-2.4.0-bin-hadoop2.7.tgz spark-2.4.0-bin-hadoop2.7/bin/pyspark Python 2.7.10 (default, Aug 17 2018, 17:41:52) [GCC 4.2.1 Compatible Apple LLVM 10.0.0 (clang-1000.0.42)] on darwin ... Using Python version 2.7.10 (default, Aug 17 2018 17:41:52) SparkSession available as 'spark'. >>> Getting started with Spark - pyspark
- 13. spark-2.4.0-bin-hadoop2.7/bin/spark-shell ... Welcome to ____ __ / __/__ ___ _____/ /__ _ / _ / _ `/ __/ '_/ /___/ .__/_,_/_/ /_/_ version 2.4.0 /_/ ... scala> Getting started with Spark - spark-shell (scala)
- 14. Spark Stack
- 16. Spark Stack Spark SQL Spark Streaming MLIB GraphX Apache Spark Seamless integration with SQL using DataFrame API. Also supports HIVE SQL Fast Feed data processing API. Designed for Fault Tolerance and bridges streaming with batch processing MLib is Spark machine learning algorithms trick bag. Spark graph library
- 25. How can we use Spark with MongoDB?
- 27. Parallelism Machine Learning Stream Processing Aggregation Native Processing Horizontal Scalling
- 30. 30 What do we need ?
- 32. 32 MongoDB Spark Connector Spark SQL Spark Streaming Apache Spark MongoDB Spark Connector
- 33. 33 Input Cluster Spark SQL Spark Streaming Apache Spark MongoDB Spark Connector
- 34. 34 Output Cluster Spark SQL Spark Streaming Apache Spark MongoDB Spark Connector
- 35. 35 Which can be the same! Spark SQL Spark Streaming Apache Spark MongoDB Spark Connector
- 37. What we can do with Spark and MongoDB
- 38. Spark SQL
- 41. Fraud Detection I'm so in love!
- 42. Fraud Detection I'm so in love! Me, too<3 Now send me your CC number ? Ok, XXXX-123-zzz $$$
- 43. Fraud Detection
- 44. Workloads Chat App Login User Profile Contacts Messages … Spark Fraud Detection Segmentation Recommendations HDFS HDFS HDFS Archiving Data Crunching
- 45. Workloads Chat App Spark Real-time data processing HDFS HDFS HDFS
- 46. Spark Streaming
- 48. 48 Spark Streaming Twitter Feed { "statuses": [ { "coordinates": null, "favorited": false, "truncated": false, "created_at": "Mon Sep 24 03:35:21 +0000 2012", "id_str": "250075927172759552", "entities": { "urls": [ ], "hashtags": [ { "text": "freebandnames", "indices": [ 20, 34 ] } ], "user_mentions": [] } } }
- 49. 49 Spark Streaming Spark { "statuses": [ { "coordinates": null, "favorited": false, "truncated": false, "created_at": "Mon Sep 24 03:35:21 +0000 2012", "id_str": "250075927172759552", "entities": { "urls": [ ], "hashtags": [ { "text": "freebandnames", "indices": [ 20, 34 ] } ], "user_mentions": [] } } } { "time": "Mon Sep 24 03:35", "freebandnames": 1 } { "statuses": [ { "coordinates": null, "favorited": false, "truncated": false, "created_at": "Mon Sep 24 03:35:21 +0000 2012", "id_str": "250075927172759552", "entities": { "urls": [ ], "hashtags": [ { "text": "freebandnames", "indices": [ 20, 34 ] } ], "user_mentions": [] } } } { "statuses": [ { "coordinates": null, "favorited": false, "truncated": false, "created_at": "Mon Sep 24 03:35:21 +0000 2012", "id_str": "250075927172759552", "entities": { "urls": [ ], "hashtags": [ { "text": "freebandnames", "indices": [ 20, 34 ] } ], "user_mentions": [] } } } { "statuses": [ { "coordinates": null, "favorited": false, "truncated": false, "created_at": "Mon Sep 24 03:35:21 +0000 2012", "id_str": "250075927172759552", "entities": { "urls": [ ], "hashtags": [ { "text": "freebandnames", "indices": [ 20, 34 ] } ], "user_mentions": [] } } } { "time": "Mon Sep 24 03:35", "freebandnames": 4 }
- 50. Clear, right?
- 51. Demo Time
- 54. 54 Steps Describing Demo ● SRT text messages in the network ● Spark collects those messages ○ Defines a processing Window ○ Performs word count ● Store DataFrame into MongoDB https://github.com/nleite/mdb.local
- 55. In Short
- 56. 56 ● An extremely powerful combination ● Many possible use cases ● Evolving all the time Spark and MongoDB