SlideShare a Scribd company logo

How Apache Spark fits in the Big Data landscape

Paco Nathan
Paco Nathan

Apache Spark, developed at UC Berkeley and open sourced in 2010, serves as a leading framework in the big data landscape, offering advantages like in-memory computing and support for advanced analytics on Hadoop clusters. Its core features include ease of development with native APIs, a generalized execution model, and capabilities for SQL, streaming, machine learning, and graph processing in a unified environment. The document discusses Spark's growth, historical context, and technical distinctions from other big data processing systems like MapReduce.

Technology
1 of 83
Downloaded 159 times
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
76
77
78
79
80
81
82
83
How Apache Spark fits into the Big Data landscape Stockholm Big Data 2014-11-13 meetup.com/The-Stockholm-Big-Data-Group/events/212782912/ Paco Nathan @pacoid 1
What is Spark? 2
What is Spark? Developed in 2009 at UC Berkeley AMPLab, then open sourced in 2010, Spark has since become one of the largest OSS communities in big data, with over 200 contributors in 50+ organizations spark.apache.org “Organizations that are looking at big data challenges – including collection, ETL, storage, exploration and analytics – should consider Spark for its in-memory performance and the breadth of its model. It supports advanced analytics solutions on Hadoop clusters, including the iterative model required for machine learning and graph analysis.” Gartner, Advanced Analytics and Data Science (2014) 3
What is Spark? 4
What is Spark? Spark Core is the general execution engine for the Spark platform that other functionality is built atop: ! • in-memory computing capabilities deliver speed • general execution model supports wide variety of use cases • ease of development – native APIs in Java, Scala, Python (+ SQL, Clojure, R) 5
What is Spark? WordCount in 3 lines of Spark WordCount in 50+ lines of Java MR 6
What is Spark? Sustained exponential growth, as one of the most active Apache projects ohloh.net/orgs/apache 7
TL;DR: Smashing The Previous Petabyte Sort Record databricks.com/blog/2014/11/05/spark-officially-sets- a-new-record-in-large-scale-sorting.html 8
A Brief History 9
A Brief History: Functional Programming for Big Data Theory, Eight Decades Ago: what can be computed? Haskell Curry haskell.org Alonso Church wikipedia.org Praxis, Four Decades Ago: algebra for applicative systems John Backus acm.org David Turner wikipedia.org Reality, Two Decades Ago: machine data from web apps Pattie Maes MIT Media Lab 10
A Brief History: Functional Programming for Big Data circa late 1990s: explosive growth e-commerce and machine data implied that workloads could not fit on a single computer anymore… notable firms led the shift to horizontal scale-out on clusters of commodity hardware, especially for machine learning use cases at scale 11
Stakeholder Customers RDBMS SQL Query result sets recommenders + classifiers Web Apps customer transactions Algorithmic Modeling Logs event history aggregation dashboards Product Engineering UX DW ETL Middleware models servlets 12
Amazon “Early Amazon: Splitting the website” – Greg Linden glinden.blogspot.com/2006/02/early-amazon-splitting- website.html ! eBay “The eBay Architecture” – Randy Shoup, Dan Pritchett addsimplicity.com/adding_simplicity_an_engi/ 2006/11/you_scaled_your.html addsimplicity.com.nyud.net:8080/downloads/ eBaySDForum2006-11-29.pdf ! Inktomi (YHOO Search) “Inktomi’s Wild Ride” – Erik Brewer (0:05:31 ff) youtu.be/E91oEn1bnXM ! Google “Underneath the Covers at Google” – Jeff Dean (0:06:54 ff) youtu.be/qsan-GQaeyk perspectives.mvdirona.com/2008/06/11/ JeffDeanOnGoogleInfrastructure.aspx ! MIT Media Lab “Social Information Filtering for Music Recommendation” – Pattie Maes pubs.media.mit.edu/pubs/papers/32paper.ps ted.com/speakers/pattie_maes.html 13
A Brief History: Functional Programming for Big Data circa 2002: mitigate risk of large distributed workloads lost due to disk failures on commodity hardware… Google File System Sanjay Ghemawat, Howard Gobioff, Shun-Tak Leung research.google.com/archive/gfs.html ! MapReduce: Simplified Data Processing on Large Clusters Jeffrey Dean, Sanjay Ghemawat research.google.com/archive/mapreduce.html 14
A Brief History: Functional Programming for Big Data 2002 2004 MapReduce paper 2002 MapReduce @ Google 2004 2006 2008 2010 2012 2014 2006 Hadoop @ Yahoo! 2014 Apache Spark top-level 2010 Spark paper 2008 Hadoop Summit 15
TL;DR: Generational trade-offs for handling Big Compute Photo from John Wilkes’ keynote talk @ #MesosCon 2014 16
TL;DR: Generational trade-offs for handling Big Compute Cheap Memory Cheap Storage Cheap Network recompute replicate reference (RDD) (DFS) (URI) 17
A Brief History: Functional Programming for Big Data MapReduce Pregel Giraph Dremel Drill S4 Storm F1 MillWheel General Batch Processing Specialized Systems: Impala GraphLab iterative, interactive, streaming, graph, etc. Tez MR doesn’t compose well for large applications, and so specialized systems emerged as workarounds 18
A Brief History: Functional Programming for Big Data circa 2010: a unified engine for enterprise data workflows, based on commodity hardware a decade later… Spark: Cluster Computing with Working Sets Matei Zaharia, Mosharaf Chowdhury, Michael Franklin, Scott Shenker, Ion Stoica people.csail.mit.edu/matei/papers/2010/hotcloud_spark.pdf ! Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing Matei Zaharia, Mosharaf Chowdhury, Tathagata Das, Ankur Dave, Justin Ma, Murphy McCauley, Michael Franklin, Scott Shenker, Ion Stoica usenix.org/system/files/conference/nsdi12/nsdi12-final138.pdf 19
A Brief History: Functional Programming for Big Data In addition to simple map and reduce operations, Spark supports SQL queries, streaming data, and complex analytics such as machine learning and graph algorithms out-of-the-box. Better yet, combine these capabilities seamlessly into one integrated workflow… 20
A Brief History: Key distinctions for Spark vs. MapReduce • generalized patterns ⇒ unified engine for many use cases • lazy evaluation of the lineage graph ⇒ reduces wait states, better pipelining • generational differences in hardware ⇒ off-heap use of large memory spaces • functional programming / ease of use ⇒ reduction in cost to maintain large apps • lower overhead for starting jobs • less expensive shuffles 21
TL;DR: Engineering is about costs Sure, maybe you’ll squeeze slightly better performance by using many specialized systems… However, putting on an Eng Director hat, would you be also prepared to pay the corresponding costs of: • learning curves for your developers across several different frameworks • ops for several different kinds of clusters • maintenance + troubleshooting mission-critical apps across several systems • tech-debt for OSS that ignores the math (80 yrs!) plus the fundamental h/w trade-offs 22
Spark Deconstructed 23
Spark Deconstructed: Log Mining Example // load error messages from a log into memory! // then interactively search for various patterns! // https://gist.github.com/ceteri/8ae5b9509a08c08a1132! ! // base RDD! val lines = sc.textFile("hdfs://...")! ! // transformed RDDs! val errors = lines.filter(_.startsWith("ERROR"))! val messages = errors.map(_.split("t")).map(r => r(1))! messages.cache()! ! // action 1! messages.filter(_.contains("mysql")).count()! ! // action 2! messages.filter(_.contains("php")).count() 24
Driver Worker Worker Worker Spark Deconstructed: Log Mining Example We start with Spark running on a cluster… submitting code to be evaluated on it: 25
Spark Deconstructed: Log Mining Example // base RDD! val lines = sc.textFile("hdfs://...")! ! // transformed RDDs! val errors = lines.filter(_.startsWith("ERROR"))! val messages = errors.map(_.split("t")).map(r => r(1))! messages.cache()! ! // action 1! messages.filter(_.contains("mysql")).count()! ! // discussing action 2! the other part messages.filter(_.contains("php")).count() 26
Spark Deconstructed: Log Mining Example At this point, take a look at the transformed RDD operator graph: scala> messages.toDebugString! res5: String = ! MappedRDD[4] at map at <console>:16 (3 partitions)! MappedRDD[3] at map at <console>:16 (3 partitions)! FilteredRDD[2] at filter at <console>:14 (3 partitions)! MappedRDD[1] at textFile at <console>:12 (3 partitions)! HadoopRDD[0] at textFile at <console>:12 (3 partitions) 27
Driver Worker Worker Worker Spark Deconstructed: Log Mining Example // base RDD! val lines = sc.textFile("hdfs://...")! ! // transformed RDDs! val errors = lines.filter(_.startsWith("ERROR"))! val messages = errors.map(_.split("t")).map(r => r(1))! messages.cache()! ! // action 1! messages.filter(_.contains("mysql")).count()! ! // action 2! medssaigsesc.fuilstesr(i_n.cognt atinhs(e"ph po")t).hcoeuntr() part 28
Driver Worker Worker block 1 Worker block 2 block 3 Spark Deconstructed: Log Mining Example // base RDD! val lines = sc.textFile("hdfs://...")! ! // transformed RDDs! val errors = lines.filter(_.startsWith("ERROR"))! val messages = errors.map(_.split("t")).map(r => r(1))! messages.cache()! ! // action 1! messages.filter(_.contains("mysql")).count()! ! // action 2! medssaigsesc.fuilstesr(i_n.cognt atinhs(e"ph po")t).hcoeuntr() part 29
Driver Worker Worker block 1 Worker block 2 block 3 Spark Deconstructed: Log Mining Example // base RDD! val lines = sc.textFile("hdfs://...")! ! // transformed RDDs! val errors = lines.filter(_.startsWith("ERROR"))! val messages = errors.map(_.split("t")).map(r => r(1))! messages.cache()! ! // action 1! messages.filter(_.contains("mysql")).count()! ! // action 2! medssaigsesc.fuilstesr(i_n.cognt atinhs(e"ph po")t).hcoeuntr() part 30
Driver Worker Worker block 1 Worker block 2 block 3 read HDFS block read HDFS block read HDFS block Spark Deconstructed: Log Mining Example // base RDD! val lines = sc.textFile("hdfs://...")! ! // transformed RDDs! val errors = lines.filter(_.startsWith("ERROR"))! val messages = errors.map(_.split("t")).map(r => r(1))! messages.cache()! ! // action 1! messages.filter(_.contains("mysql")).count()! ! // action 2! medssaigsesc.fuilstesr(i_n.cognt atinhs(e"ph po")t).hcoeuntr() part 31
Driver cache 1 Worker Worker block 1 Worker block 2 block 3 cache 2 cache 3 process, cache data process, cache data process, cache data Spark Deconstructed: Log Mining Example // base RDD! val lines = sc.textFile("hdfs://...")! ! // transformed RDDs! val errors = lines.filter(_.startsWith("ERROR"))! val messages = errors.map(_.split("t")).map(r => r(1))! messages.cache()! ! // action 1! messages.filter(_.contains("mysql")).count()! ! // action 2! medssaigsesc.fuilstesr(i_n.cognt atinhs(e"ph po")t).hcoeuntr() part 32
Driver cache 1 Worker Worker block 1 Worker block 2 block 3 cache 2 cache 3 Spark Deconstructed: Log Mining Example // base RDD! val lines = sc.textFile("hdfs://...")! ! // transformed RDDs! val errors = lines.filter(_.startsWith("ERROR"))! val messages = errors.map(_.split("t")).map(r => r(1))! messages.cache()! ! // action 1! messages.filter(_.contains("mysql")).count()! ! // action 2! medssaigsesc.fuilstesr(i_n.cognt atinhs(e"ph po")t).hcoeuntr() part 33
// base RDD! val lines = sc.textFile("hdfs://...")! ! // transformed RDDs! val errors = lines.filter(_.startsWith("ERROR"))! val messages = errors.map(_.split("t")).map(r => r(1))! messages.cache()! ! // action 1! messages.filter(_.contains("mysql")).count()! ! // action 2! messages.filter(_.contains("php")).count() Driver cache 1 Worker Worker block 1 Worker block 2 block 3 cache 2 cache 3 Spark Deconstructed: Log Mining Example discussing the other part 34
Driver cache 1 Worker Worker block 1 Worker block 2 block 3 cache 2 cache 3 process from cache process from cache process from cache Spark Deconstructed: Log Mining Example // base RDD! val lines = sc.textFile("hdfs://...")! ! // discussing transformed RDDs! val errors = lines.filter(_.the startsWith("other ERROR"))part ! val messages = errors.map(_.split("t")).map(r => r(1))! messages.cache()! ! // action 1! messages.filter(_.contains(“mysql")).count()! ! // action 2! messages.filter(_.contains("php")).count() 35
Driver cache 1 Worker Worker block 1 Worker block 2 block 3 cache 2 cache 3 Spark Deconstructed: Log Mining Example // base RDD! val lines = sc.textFile("hdfs://...")! ! // discussing transformed RDDs! val errors = lines.filter(_.the startsWith("other ERROR"))part ! val messages = errors.map(_.split("t")).map(r => r(1))! messages.cache()! ! // action 1! messages.filter(_.contains(“mysql")).count()! ! // action 2! messages.filter(_.contains("php")).count() 36
Spark Deconstructed: Looking at the RDD transformations and actions from another perspective… action value RDD RDD RDD // load error messages from a log into memory! // then interactively search for various patterns! // https://gist.github.com/ceteri/8ae5b9509a08c08a1132! ! // base RDD! val lines = sc.textFile("hdfs://...")! ! // transformed RDDs! val errors = lines.filter(_.startsWith("ERROR"))! val messages = errors.map(_.split("t")).map(r => r(1))! messages.cache()! ! // action 1! messages.filter(_.contains("mysql")).count()! ! // action 2! messages.filter(_.contains("php")).count() transformations RDD 37
Spark Deconstructed: RDD // base RDD! val lines = sc.textFile("hdfs://...") 38
RDD RDD RDD Spark Deconstructed: transformations RDD // transformed RDDs! val errors = lines.filter(_.startsWith("ERROR"))! val messages = errors.map(_.split("t")).map(r => r(1))! messages.cache() 39
action value RDD RDD RDD Spark Deconstructed: transformations RDD // action 1! messages.filter(_.contains("mysql")).count() 40
Unifying the Pieces 41
Unifying the Pieces: Spark SQL // http://spark.apache.org/docs/latest/sql-programming-guide.html! ! val sqlContext = new org.apache.spark.sql.SQLContext(sc)! import sqlContext._! ! // define the schema using a case class! case class Person(name: String, age: Int)! ! // create an RDD of Person objects and register it as a table! val people = sc.textFile("examples/src/main/resources/ people.txt").map(_.split(",")).map(p => Person(p(0), p(1).trim.toInt))! ! people.registerAsTempTable("people")! ! // SQL statements can be run using the SQL methods provided by sqlContext! val teenagers = sql("SELECT name FROM people WHERE age >= 13 AND age <= 19")! ! // results of SQL queries are SchemaRDDs and support all the ! // normal RDD operations…! // columns of a row in the result can be accessed by ordinal! teenagers.map(t => "Name: " + t(0)).collect().foreach(println) 42
Unifying the Pieces: Spark Streaming // http://spark.apache.org/docs/latest/streaming-programming-guide.html! ! import org.apache.spark.streaming._! import org.apache.spark.streaming.StreamingContext._! ! // create a StreamingContext with a SparkConf configuration! val ssc = new StreamingContext(sparkConf, Seconds(10))! ! // create a DStream that will connect to serverIP:serverPort! val lines = ssc.socketTextStream(serverIP, serverPort)! ! // split each line into words! val words = lines.flatMap(_.split(" "))! ! // count each word in each batch! val pairs = words.map(word => (word, 1))! val wordCounts = pairs.reduceByKey(_ + _)! ! // print a few of the counts to the console! wordCounts.print()! ! ssc.start() // start the computation! ssc.awaitTermination() // wait for the computation to terminate 43
MLI: An API for Distributed Machine Learning Evan Sparks, Ameet Talwalkar, et al. International Conference on Data Mining (2013) http://arxiv.org/abs/1310.5426 Unifying the Pieces: MLlib // http://spark.apache.org/docs/latest/mllib-guide.html! ! val train_data = // RDD of Vector! val model = KMeans.train(train_data, k=10)! ! // evaluate the model! val test_data = // RDD of Vector! test_data.map(t => model.predict(t)).collect().foreach(println)! 44
Unifying the Pieces: GraphX // http://spark.apache.org/docs/latest/graphx-programming-guide.html! ! import org.apache.spark.graphx._! import org.apache.spark.rdd.RDD! ! case class Peep(name: String, age: Int)! ! val vertexArray = Array(! (1L, Peep("Kim", 23)), (2L, Peep("Pat", 31)),! (3L, Peep("Chris", 52)), (4L, Peep("Kelly", 39)),! (5L, Peep("Leslie", 45))! )! val edgeArray = Array(! Edge(2L, 1L, 7), Edge(2L, 4L, 2),! Edge(3L, 2L, 4), Edge(3L, 5L, 3),! Edge(4L, 1L, 1), Edge(5L, 3L, 9)! )! ! val vertexRDD: RDD[(Long, Peep)] = sc.parallelize(vertexArray)! val edgeRDD: RDD[Edge[Int]] = sc.parallelize(edgeArray)! val g: Graph[Peep, Int] = Graph(vertexRDD, edgeRDD)! ! val results = g.triplets.filter(t => t.attr > 7)! ! for (triplet <- results.collect) {! println(s"${triplet.srcAttr.name} loves ${triplet.dstAttr.name}")! } 45
Demos, as time permits: brand new Python support for Streaming in 1.2 github.com/apache/spark/tree/master/examples/src/main/ python/streaming Twitter Streaming Language Classifier databricks.gitbooks.io/databricks-spark-reference-applications/ content/twitter_classifier/README.html ! ! For more Spark learning resources online: databricks.com/spark-training-resources 46
Complementary Frameworks 47
Spark Integrations: Discover Insights Clean Up Your Data Run Sophisticated Analytics Integrate With Many Other Systems Use Lots of Different Data Sources cloud-based notebooks… ETL… the Hadoop ecosystem… widespread use of PyData… advanced analytics in streaming… rich custom search… web apps for data APIs… low-latency + multi-tenancy… 48
Spark Integrations: Unified platform for building Big Data pipelines Databricks Cloud databricks.com/blog/2014/07/14/databricks-cloud-making- big-data-easy.html youtube.com/watch?v=dJQ5lV5Tldw#t=883 49
Spark Integrations: The proverbial Hadoop ecosystem Spark + Hadoop + HBase + etc. mapr.com/products/apache-spark vision.cloudera.com/apache-spark-in-the-apache-hadoop-ecosystem/ hortonworks.com/hadoop/spark/ databricks.com/blog/2014/05/23/pivotal-hadoop-integrates-the- full-apache-spark-stack.html unified compute hadoop ecosystem 50
Spark Integrations: Leverage widespread use of Python Spark + PyData spark-summit.org/2014/talk/A-platform-for-large-scale-neuroscience cwiki.apache.org/confluence/display/SPARK/PySpark+Internals unified compute Py Data 51
Spark Integrations: Advanced analytics for streaming use cases Kafka + Spark + Cassandra datastax.com/documentation/datastax_enterprise/4.5/ datastax_enterprise/spark/sparkIntro.html http://helenaedelson.com/?p=991 github.com/datastax/spark-cassandra-connector github.com/dibbhatt/kafka-spark-consumer unified compute data streams columnar key-value 52
Spark Integrations: Rich search, immediate insights Spark + ElasticSearch databricks.com/blog/2014/06/27/application-spotlight-elasticsearch. unified compute html elasticsearch.org/guide/en/elasticsearch/hadoop/current/ spark.html spark-summit.org/2014/talk/streamlining-search-indexing- using-elastic-search-and-spark document search 53
Spark Integrations: Building data APIs with web apps Spark + Play typesafe.com/blog/apache-spark-and-the-typesafe-reactive- platform-a-match-made-in-heaven unified compute web apps 54
Spark Integrations: The case for multi-tenancy Spark + Mesos spark.apache.org/docs/latest/running-on-mesos.html + Mesosphere + Google Cloud Platform ceteri.blogspot.com/2014/09/spark-atop-mesos-on-google-cloud.html unified compute cluster resources 55
Because Use Cases 56
Spark at Twitter: Evaluation & Lessons Learnt Sriram Krishnan slideshare.net/krishflix/seattle-spark-meetup-spark- at-twitter • Spark can be more interactive, efficient than MR • Support for iterative algorithms and caching • More generic than traditional MapReduce • Why is Spark faster than Hadoop MapReduce? • Fewer I/O synchronization barriers • Less expensive shuffle • More complex the DAG, greater the performance improvement 57 Because Use Cases: Twitter
Using Spark to Ignite Data Analytics ebaytechblog.com/2014/05/28/using-spark-to-ignite- data-analytics/ 58 Because Use Cases: eBay/PayPal
Hadoop and Spark Join Forces in Yahoo Andy Feng spark-summit.org/talk/feng-hadoop-and-spark-join- forces-at-yahoo/ 59 Because Use Cases: Yahoo!
Collaborative Filtering with Spark Chris Johnson slideshare.net/MrChrisJohnson/collaborative-filtering- with-spark • collab filter (ALS) for music recommendation • Hadoop suffers from I/O overhead • show a progression of code rewrites, converting a Hadoop-based app into efficient use of Spark 60 Because Use Cases: Spotify
Because Use Cases: Sharethrough Sharethrough Uses Spark Streaming to Optimize Bidding in Real Time Russell Cardullo, Michael Ruggier 2014-03-25 databricks.com/blog/2014/03/25/ sharethrough-and-spark-streaming.html • the profile of a 24 x 7 streaming app is different than an hourly batch job… • take time to validate output against the input… • confirm that supporting objects are being serialized… • the output of your Spark Streaming job is only as reliable as the queue that feeds Spark… • monoids… 61
Because Use Cases: Ooyala Productionizing a 24/7 Spark Streaming service on YARN Issac Buenrostro, Arup Malakar 2014-06-30 spark-summit.org/2014/talk/ productionizing-a-247-spark-streaming-service- on-yarn • state-of-the-art ingestion pipeline, processing over two billion video events a day • how do you ensure 24/7 availability and fault tolerance? • what are the best practices for Spark Streaming and its integration with Kafka and YARN? • how do you monitor and instrument the various 62 stages of the pipeline?
Because Use Cases: Viadeo Spark Streaming As Near Realtime ETL Djamel Zouaoui 2014-09-18 slideshare.net/DjamelZouaoui/spark-streaming • Spark Streaming is topology-free • workers and receivers are autonomous and independent • paired with Kafka, RabbitMQ • 8 machines / 120 cores • use case for recommender system • issues: how to handle lost data, serialization 63
Because Use Cases: Stratio Stratio Streaming: a new approach to Spark Streaming David Morales, Oscar Mendez 2014-06-30 spark-summit.org/2014/talk/stratio-streaming- a-new-approach-to-spark-streaming • Stratio Streaming is the union of a real-time messaging bus with a complex event processing engine using Spark Streaming • allows the creation of streams and queries on the fly • paired with Siddhi CEP engine and Apache Kafka • added global features to the engine such as auditing 64 and statistics
Because Use Cases: Guavus Guavus Embeds Apache Spark into its Operational Intelligence Platform Deployed at the World’s Largest Telcos Eric Carr 2014-09-25 databricks.com/blog/2014/09/25/guavus-embeds-apache-spark-into- its-operational-intelligence-platform-deployed-at-the-worlds- largest-telcos.html • 4 of 5 top mobile network operators, 3 of 5 top Internet backbone providers, 80% MSOs in NorAm • analyzing 50% of US mobile data traffic, +2.5 PB/day • latency is critical for resolving operational issues before they cascade: 2.5 MM transactions per second • “analyze first” not “store first ask questions later” 65
Why Spark is the Next Top (Compute) Model Dean Wampler slideshare.net/deanwampler/spark-the-next-top- compute-model • Hadoop: most algorithms are much harder to implement in this restrictive map-then-reduce model • Spark: fine-grained “combinators” for composing algorithms • slide #67, any questions? 66 Because Use Cases: Typesafe
Installing the Cassandra / Spark OSS Stack Al Tobey tobert.github.io/post/2014-07-15-installing-cassandra- spark-stack.html • install+config for Cassandra and Spark together • spark-cassandra-connector integration • examples show a Spark shell that can access tables in Cassandra as RDDs with types pre-mapped and ready to go 67 Because Use Cases: DataStax
One platform for all: real-time, near-real-time, and offline video analytics on Spark Davis Shepherd, Xi Liu spark-summit.org/talk/one-platform-for-all-real- time-near-real-time-and-offline-video-analytics- on-spark 68 Because Use Cases: Conviva
Approximations (appended during talk) 69
Approximations 19-20c. statistics emphasized defensibility in lieu of predictability, based on analytic variance and goodness-of-fit tests ! That approach inherently led toward a manner of computational thinking based on batch windows ! They missed a subtle point… 70
21c. shift towards modeling based on probabilistic approximations: trade bounded errors for greatly reduced resource costs highlyscalable.wordpress.com/2012/05/01/ probabilistic-structures-web-analytics-data- mining/ 71 Approximations
21c. shift towards modeling based on probabil approximations: trade bounded errors for greatly reduced resource costs Twitter catch-phrase: “Hash, don’t sample” highlyscalable.wordpress.com/2012/05/01/ probabilistic-structures-web-analytics-data- mining/ 72 Approximations
a fascinating and relatively new area, pioneered by relatively few people – e.g., Philippe Flajolet provides approximation, with error bounds – in general uses significantly less resources (RAM, CPU, etc.) many algorithms can be constructed from combinations of read and write monoids aggregate different ranges by composing hashes, instead of repeating full-queries 73 Approximations
algorithm use case example Count-Min Sketch frequency summaries code HyperLogLog set cardinality code Bloom Filter set membership MinHash set similarity DSQ streaming quantiles SkipList ordered sequence search 74 Approximations
algorithm use case example Count-Min Sketch frequency summaries code HyperLogLog set cardinality code suggestion: consider these as your most quintessential collections data types at scale Bloom Filter set membership MinHash set similarity DSQ streaming quantiles SkipList ordered sequence search 75 Approximations
• sketch algorithms: trade bounded errors for orders of magnitude less required resources, e.g., fit more complex apps in memory • multicore + large memory spaces (off heap) are increasing the resources per node in a cluster • containers allow for finer-grain allocation of cluster resources and multi-tenancy • monoids, etc.: guarantees of associativity within the code allow for more effective distributed computing, e.g., partial aggregates • less resources must be spent sorting/windowing data prior to working with a data set • real-time apps, which don’t have the luxury of anticipating data partitions, can respond quickly 76 Approximations
Probabilistic Data Structures for Web Analytics and Data Mining Ilya Katsov (2012-05-01) A collection of links for streaming algorithms and data structures Debasish Ghosh Aggregate Knowledge blog (now Neustar) Timon Karnezos, Matt Curcio, et al. Probabilistic Data Structures and Breaking Down Big Sequence Data C. Titus Brown, O'Reilly (2010-11-10) Algebird Avi Bryant, Oscar Boykin, et al. Twitter (2012) Mining of Massive Datasets Jure Leskovec, Anand Rajaraman, Jeff Ullman, Cambridge (2011) 77 Approximations
Resources 78
certification: Apache Spark developer certificate program • http://oreilly.com/go/sparkcert • defined by Spark experts @Databricks • assessed by O’Reilly Media • establishes the bar for Spark expertise 79
community: spark.apache.org/community.html video+slide archives: spark-summit.org events worldwide: goo.gl/2YqJZK resources: databricks.com/spark-training-resources workshops: databricks.com/spark-training Intro to Spark Spark AppDev Spark DevOps Spark DataSci Distributed ML on Spark Streaming Apps on Spark Spark + Cassandra 80
books: Fast Data Processing with Spark Holden Karau Packt (2013) shop.oreilly.com/product/ 9781782167068.do Spark in Action Chris Fregly Manning (2015*) sparkinaction.com/ Learning Spark Holden Karau, Andy Konwinski, Matei Zaharia O’Reilly (2015*) shop.oreilly.com/product/ 0636920028512.do 81
events: Big Data Spain Madrid, Nov 17-18 bigdataspain.org Strata EU Barcelona, Nov 19-21 strataconf.com/strataeu2014 Data Day Texas Austin, Jan 10 datadaytexas.com Strata CA San Jose, Feb 18-20 strataconf.com/strata2015 Spark Summit East NYC, Mar 18-19 spark-summit.org/east Strata EU London, May 5-7 strataconf.com/big-data-conference-uk-2015 Spark Summit 2015 SF, Jun 15-17 spark-summit.org 82
presenter: monthly newsletter for updates, events, conf summaries, etc.: liber118.com/pxn/ Just Enough Math O’Reilly, 2014 justenoughmath.com preview: youtu.be/TQ58cWgdCpA Enterprise Data Workflows with Cascading O’Reilly, 2013 shop.oreilly.com/product/ 0636920028536.do 83

More Related Content

What's hot (20)

PDF
Spark streaming
Noam Shaish
 
PDF
Data Science with Spark
Krishna Sankar
 
PDF
Graph Analytics in Spark
Paco Nathan
 
PDF
Functional programming
 for optimization problems 
in Big Data
Paco Nathan
 
PDF
Introduction to Spark Training
Spark Summit
 
PDF
Adding Complex Data to Spark Stack-(Neeraja Rentachintala, MapR)
Spark Summit
 
PDF
An excursion into Graph Analytics with Apache Spark GraphX
Krishna Sankar
 
PPTX
AMP Camp 5 Intro
jeykottalam
 
PDF
H2O Big Join Slides
Sri Ambati
 
PDF
Spark Summit East 2015 Keynote -- Databricks CEO Ion Stoica
Databricks
 
PPTX
Big Data Analytics with Storm, Spark and GraphLab
Impetus Technologies
 
PDF
Big data apache spark + scala
Juantomás García Molina
 
PDF
Spark + H20 = Machine Learning at scale
Mateusz Dymczyk
 
PDF
GraphFrames: DataFrame-based graphs for Apache® Spark™
Databricks
 
PPTX
JEEConf 2015 - Introduction to real-time big data with Apache Spark
Taras Matyashovsky
 
PDF
PDX Hadoop: Enterprise Data Workflows with Cascading and Mesos
Paco Nathan
 
PDF
New Directions for Spark in 2015 - Spark Summit East
Databricks
 
PDF
Dynamic Community Detection for Large-scale e-Commerce data with Spark Stream...
Spark Summit
 
PDF
Stratio CrossData: an efficient distributed datahub with batch and streaming ...
Stratio
 
PDF
Composable Parallel Processing in Apache Spark and Weld
Databricks
 
Spark streaming
Noam Shaish
 
Data Science with Spark
Krishna Sankar
 
Graph Analytics in Spark
Paco Nathan
 
Functional programming
 for optimization problems 
in Big Data
Paco Nathan
 
Introduction to Spark Training
Spark Summit
 
Adding Complex Data to Spark Stack-(Neeraja Rentachintala, MapR)
Spark Summit
 
An excursion into Graph Analytics with Apache Spark GraphX
Krishna Sankar
 
AMP Camp 5 Intro
jeykottalam
 
H2O Big Join Slides
Sri Ambati
 
Spark Summit East 2015 Keynote -- Databricks CEO Ion Stoica
Databricks
 
Big Data Analytics with Storm, Spark and GraphLab
Impetus Technologies
 
Big data apache spark + scala
Juantomás García Molina
 
Spark + H20 = Machine Learning at scale
Mateusz Dymczyk
 
GraphFrames: DataFrame-based graphs for Apache® Spark™
Databricks
 
JEEConf 2015 - Introduction to real-time big data with Apache Spark
Taras Matyashovsky
 
PDX Hadoop: Enterprise Data Workflows with Cascading and Mesos
Paco Nathan
 
New Directions for Spark in 2015 - Spark Summit East
Databricks
 
Dynamic Community Detection for Large-scale e-Commerce data with Spark Stream...
Spark Summit
 
Stratio CrossData: an efficient distributed datahub with batch and streaming ...
Stratio
 
Composable Parallel Processing in Apache Spark and Weld
Databricks
 

Viewers also liked (19)

PDF
Data Science in 2016: Moving Up
Paco Nathan
 
PDF
Data Science Reinvents Learning?
Paco Nathan
 
PDF
GalvanizeU Seattle: Eleven Almost-Truisms About Data
Paco Nathan
 
PDF
Microservices, Containers, and Machine Learning
Paco Nathan
 
PDF
GraphX: Graph analytics for insights about developer communities
Paco Nathan
 
PDF
A New Year in Data Science: ML Unpaused
Paco Nathan
 
PDF
QCon São Paulo: Real-Time Analytics with Spark Streaming
Paco Nathan
 
PDF
Jupyter for Education: Beyond Gutenberg and Erasmus
Paco Nathan
 
PDF
How Apache Spark fits into the Big Data landscape
Paco Nathan
 
PDF
SF Python Meetup: TextRank in Python
Paco Nathan
 
PDF
[ICIST 2013] A Probabilistic Relational Data Model for Uncertain Information
Hieu Tran
 
PDF
Advanced Apache Spark Meetup Approximations and Probabilistic Data Structures...
Chris Fregly
 
PDF
Data Science in Future Tense
Paco Nathan
 
PDF
#MesosCon 2014: Spark on Mesos
Paco Nathan
 
PDF
OSCON 2014: Data Workflows for Machine Learning
Paco Nathan
 
PDF
Big Data is changing abruptly, and where it is likely heading
Paco Nathan
 
PPTX
Future of data science as a profession
Jose Quesada
 
PDF
Big data & data science challenges and opportunities
Jose Quesada
 
PDF
5 Ways to Protect Your Healthcare Organization from a Ransomware Attack - HIM...
ClearDATACloud
 
Data Science in 2016: Moving Up
Paco Nathan
 
Data Science Reinvents Learning?
Paco Nathan
 
GalvanizeU Seattle: Eleven Almost-Truisms About Data
Paco Nathan
 
Microservices, Containers, and Machine Learning
Paco Nathan
 
GraphX: Graph analytics for insights about developer communities
Paco Nathan
 
A New Year in Data Science: ML Unpaused
Paco Nathan
 
QCon São Paulo: Real-Time Analytics with Spark Streaming
Paco Nathan
 
Jupyter for Education: Beyond Gutenberg and Erasmus
Paco Nathan
 
How Apache Spark fits into the Big Data landscape
Paco Nathan
 
SF Python Meetup: TextRank in Python
Paco Nathan
 
[ICIST 2013] A Probabilistic Relational Data Model for Uncertain Information
Hieu Tran
 
Advanced Apache Spark Meetup Approximations and Probabilistic Data Structures...
Chris Fregly
 
Data Science in Future Tense
Paco Nathan
 
#MesosCon 2014: Spark on Mesos
Paco Nathan
 
OSCON 2014: Data Workflows for Machine Learning
Paco Nathan
 
Big Data is changing abruptly, and where it is likely heading
Paco Nathan
 
Future of data science as a profession
Jose Quesada
 
Big data & data science challenges and opportunities
Jose Quesada
 
5 Ways to Protect Your Healthcare Organization from a Ransomware Attack - HIM...
ClearDATACloud
 
Ad

Similar to How Apache Spark fits in the Big Data landscape (20)

PDF
Brief Intro to Apache Spark @ Stanford ICME
Paco Nathan
 
PDF
Big Data Everywhere Chicago: Apache Spark Plus Many Other Frameworks -- How S...
BigDataEverywhere
 
PPTX
2016-07-21-Godil-presentation.pptx
D21CE161GOSWAMIPARTH
 
PDF
Big data processing with apache spark
sarith divakar
 
PDF
Apache Hadoop and Spark: Introduction and Use Cases for Data Analysis
Trieu Nguyen
 
PDF
Bds session 13 14
Infinity Tech Solutions
 
PPTX
In Memory Analytics with Apache Spark
Venkata Naga Ravi
 
PDF
Dev Ops Training
Spark Summit
 
PPTX
Big data vahidamiri-tabriz-13960226-datastack.ir
datastack
 
PDF
Apache Spark 101 - Demi Ben-Ari
Demi Ben-Ari
 
PPTX
Big data clustering
Jagadeesan A S
 
PPTX
Intro to Spark development
Spark Summit
 
PDF
20150716 introduction to apache spark v3
Andrey Vykhodtsev
 
PPTX
Spark.pptx to knowledge gaining in wdm days ago
PreethamMCPreethamMC
 
PPTX
Apache Spark Fundamentals
Zahra Eskandari
 
PDF
Cassandra Summit 2014: Apache Spark - The SDK for All Big Data Platforms
DataStax Academy
 
PDF
2015 01-17 Lambda Architecture with Apache Spark, NextML Conference
DB Tsai
 
PPTX
Unit II Real Time Data Processing tools.pptx
Rahul Borate
 
PDF
Started with-apache-spark
Happiest Minds Technologies
 
PDF
Big Data Open Source Tools and Trends: Enable Real-Time Business Intelligence...
Perficient, Inc.
 
Brief Intro to Apache Spark @ Stanford ICME
Paco Nathan
 
Big Data Everywhere Chicago: Apache Spark Plus Many Other Frameworks -- How S...
BigDataEverywhere
 
2016-07-21-Godil-presentation.pptx
D21CE161GOSWAMIPARTH
 
Big data processing with apache spark
sarith divakar
 
Apache Hadoop and Spark: Introduction and Use Cases for Data Analysis
Trieu Nguyen
 
Bds session 13 14
Infinity Tech Solutions
 
In Memory Analytics with Apache Spark
Venkata Naga Ravi
 
Dev Ops Training
Spark Summit
 
Big data vahidamiri-tabriz-13960226-datastack.ir
datastack
 
Apache Spark 101 - Demi Ben-Ari
Demi Ben-Ari
 
Big data clustering
Jagadeesan A S
 
Intro to Spark development
Spark Summit
 
20150716 introduction to apache spark v3
Andrey Vykhodtsev
 
Spark.pptx to knowledge gaining in wdm days ago
PreethamMCPreethamMC
 
Apache Spark Fundamentals
Zahra Eskandari
 
Cassandra Summit 2014: Apache Spark - The SDK for All Big Data Platforms
DataStax Academy
 
2015 01-17 Lambda Architecture with Apache Spark, NextML Conference
DB Tsai
 
Unit II Real Time Data Processing tools.pptx
Rahul Borate
 
Started with-apache-spark
Happiest Minds Technologies
 
Big Data Open Source Tools and Trends: Enable Real-Time Business Intelligence...
Perficient, Inc.
 
Ad

More from Paco Nathan (8)

PDF
Human in the loop: a design pattern for managing teams working with ML
Paco Nathan
 
PDF
Human-in-the-loop: a design pattern for managing teams that leverage ML
Paco Nathan
 
PDF
Human-in-a-loop: a design pattern for managing teams which leverage ML
Paco Nathan
 
PDF
Humans in a loop: Jupyter notebooks as a front-end for AI
Paco Nathan
 
PDF
Humans in the loop: AI in open source and industry
Paco Nathan
 
PDF
Computable Content
Paco Nathan
 
PDF
Computable Content: Lessons Learned
Paco Nathan
 
PDF
Strata EU 2014: Spark Streaming Case Studies
Paco Nathan
 
Human in the loop: a design pattern for managing teams working with ML
Paco Nathan
 
Human-in-the-loop: a design pattern for managing teams that leverage ML
Paco Nathan
 
Human-in-a-loop: a design pattern for managing teams which leverage ML
Paco Nathan
 
Humans in a loop: Jupyter notebooks as a front-end for AI
Paco Nathan
 
Humans in the loop: AI in open source and industry
Paco Nathan
 
Computable Content
Paco Nathan
 
Computable Content: Lessons Learned
Paco Nathan
 
Strata EU 2014: Spark Streaming Case Studies
Paco Nathan
 

Recently uploaded (20)

PDF
DevBcn - Building 10x Organizations Using Modern Productivity Metrics
Justin Reock
 
PPTX
AUTOMATION AND ROBOTICS IN PHARMA INDUSTRY.pptx
sameeraaabegumm
 
PDF
Complete JavaScript Notes: From Basics to Advanced Concepts.pdf
haydendavispro
 
PDF
Smart Air Quality Monitoring with Serrax AQM190 LITE
SERRAX TECHNOLOGIES LLP
 
PPT
Interview paper part 3, It is based on Interview Prep
SoumyadeepGhosh39
 
PPTX
Building Search Using OpenSearch: Limitations and Workarounds
Sease
 
PDF
Transcript: New from BookNet Canada for 2025: BNC BiblioShare - Tech Forum 2025
BookNet Canada
 
PDF
Complete Network Protection with Real-Time Security
L4RGINDIA
 
PDF
Human-centred design in online workplace learning and relationship to engagem...
Tracy Tang
 
PDF
Jak MŚP w Europie Środkowo-Wschodniej odnajdują się w świecie AI
dominikamizerska1
 
PPTX
UiPath Academic Alliance Educator Panels: Session 2 - Business Analyst Content
DianaGray10
 
PDF
Windsurf Meetup Ottawa 2025-07-12 - Planning Mode at Reliza.pdf
Pavel Shukhman
 
PDF
Wojciech Ciemski for Top Cyber News MAGAZINE. June 2025
Dr. Ludmila Morozova-Buss
 
PPTX
Webinar: Introduction to LF Energy EVerest
DanBrown980551
 
PDF
The Builder’s Playbook - 2025 State of AI Report.pdf
jeroen339954
 
PDF
Building Real-Time Digital Twins with IBM Maximo & ArcGIS Indoors
Safe Software
 
PDF
Persuasive AI: risks and opportunities in the age of digital debate
Speck&Tech
 
PPTX
Top iOS App Development Company in the USA for Innovative Apps
SynapseIndia
 
PPTX
OpenID AuthZEN - Analyst Briefing July 2025
David Brossard
 
PPTX
MSP360 Backup Scheduling and Retention Best Practices.pptx
MSP360
 
DevBcn - Building 10x Organizations Using Modern Productivity Metrics
Justin Reock
 
AUTOMATION AND ROBOTICS IN PHARMA INDUSTRY.pptx
sameeraaabegumm
 
Complete JavaScript Notes: From Basics to Advanced Concepts.pdf
haydendavispro
 
Smart Air Quality Monitoring with Serrax AQM190 LITE
SERRAX TECHNOLOGIES LLP
 
Interview paper part 3, It is based on Interview Prep
SoumyadeepGhosh39
 
Building Search Using OpenSearch: Limitations and Workarounds
Sease
 
Transcript: New from BookNet Canada for 2025: BNC BiblioShare - Tech Forum 2025
BookNet Canada
 
Complete Network Protection with Real-Time Security
L4RGINDIA
 
Human-centred design in online workplace learning and relationship to engagem...
Tracy Tang
 
Jak MŚP w Europie Środkowo-Wschodniej odnajdują się w świecie AI
dominikamizerska1
 
UiPath Academic Alliance Educator Panels: Session 2 - Business Analyst Content
DianaGray10
 
Windsurf Meetup Ottawa 2025-07-12 - Planning Mode at Reliza.pdf
Pavel Shukhman
 
Wojciech Ciemski for Top Cyber News MAGAZINE. June 2025
Dr. Ludmila Morozova-Buss
 
Webinar: Introduction to LF Energy EVerest
DanBrown980551
 
The Builder’s Playbook - 2025 State of AI Report.pdf
jeroen339954
 
Building Real-Time Digital Twins with IBM Maximo & ArcGIS Indoors
Safe Software
 
Persuasive AI: risks and opportunities in the age of digital debate
Speck&Tech
 
Top iOS App Development Company in the USA for Innovative Apps
SynapseIndia
 
OpenID AuthZEN - Analyst Briefing July 2025
David Brossard
 
MSP360 Backup Scheduling and Retention Best Practices.pptx
MSP360
 

How Apache Spark fits in the Big Data landscape

  • 1. How Apache Spark fits into the Big Data landscape Stockholm Big Data 2014-11-13 meetup.com/The-Stockholm-Big-Data-Group/events/212782912/ Paco Nathan @pacoid 1
  • 3. What is Spark? Developed in 2009 at UC Berkeley AMPLab, then open sourced in 2010, Spark has since become one of the largest OSS communities in big data, with over 200 contributors in 50+ organizations spark.apache.org “Organizations that are looking at big data challenges – including collection, ETL, storage, exploration and analytics – should consider Spark for its in-memory performance and the breadth of its model. It supports advanced analytics solutions on Hadoop clusters, including the iterative model required for machine learning and graph analysis.” Gartner, Advanced Analytics and Data Science (2014) 3
  • 5. What is Spark? Spark Core is the general execution engine for the Spark platform that other functionality is built atop: ! • in-memory computing capabilities deliver speed • general execution model supports wide variety of use cases • ease of development – native APIs in Java, Scala, Python (+ SQL, Clojure, R) 5
  • 6. What is Spark? WordCount in 3 lines of Spark WordCount in 50+ lines of Java MR 6
  • 7. What is Spark? Sustained exponential growth, as one of the most active Apache projects ohloh.net/orgs/apache 7
  • 8. TL;DR: Smashing The Previous Petabyte Sort Record databricks.com/blog/2014/11/05/spark-officially-sets- a-new-record-in-large-scale-sorting.html 8
  • 10. A Brief History: Functional Programming for Big Data Theory, Eight Decades Ago: what can be computed? Haskell Curry haskell.org Alonso Church wikipedia.org Praxis, Four Decades Ago: algebra for applicative systems John Backus acm.org David Turner wikipedia.org Reality, Two Decades Ago: machine data from web apps Pattie Maes MIT Media Lab 10
  • 11. A Brief History: Functional Programming for Big Data circa late 1990s: explosive growth e-commerce and machine data implied that workloads could not fit on a single computer anymore… notable firms led the shift to horizontal scale-out on clusters of commodity hardware, especially for machine learning use cases at scale 11
  • 12. Stakeholder Customers RDBMS SQL Query result sets recommenders + classifiers Web Apps customer transactions Algorithmic Modeling Logs event history aggregation dashboards Product Engineering UX DW ETL Middleware models servlets 12
  • 13. Amazon “Early Amazon: Splitting the website” – Greg Linden glinden.blogspot.com/2006/02/early-amazon-splitting- website.html ! eBay “The eBay Architecture” – Randy Shoup, Dan Pritchett addsimplicity.com/adding_simplicity_an_engi/ 2006/11/you_scaled_your.html addsimplicity.com.nyud.net:8080/downloads/ eBaySDForum2006-11-29.pdf ! Inktomi (YHOO Search) “Inktomi’s Wild Ride” – Erik Brewer (0:05:31 ff) youtu.be/E91oEn1bnXM ! Google “Underneath the Covers at Google” – Jeff Dean (0:06:54 ff) youtu.be/qsan-GQaeyk perspectives.mvdirona.com/2008/06/11/ JeffDeanOnGoogleInfrastructure.aspx ! MIT Media Lab “Social Information Filtering for Music Recommendation” – Pattie Maes pubs.media.mit.edu/pubs/papers/32paper.ps ted.com/speakers/pattie_maes.html 13
  • 14. A Brief History: Functional Programming for Big Data circa 2002: mitigate risk of large distributed workloads lost due to disk failures on commodity hardware… Google File System Sanjay Ghemawat, Howard Gobioff, Shun-Tak Leung research.google.com/archive/gfs.html ! MapReduce: Simplified Data Processing on Large Clusters Jeffrey Dean, Sanjay Ghemawat research.google.com/archive/mapreduce.html 14
  • 15. A Brief History: Functional Programming for Big Data 2002 2004 MapReduce paper 2002 MapReduce @ Google 2004 2006 2008 2010 2012 2014 2006 Hadoop @ Yahoo! 2014 Apache Spark top-level 2010 Spark paper 2008 Hadoop Summit 15
  • 16. TL;DR: Generational trade-offs for handling Big Compute Photo from John Wilkes’ keynote talk @ #MesosCon 2014 16
  • 17. TL;DR: Generational trade-offs for handling Big Compute Cheap Memory Cheap Storage Cheap Network recompute replicate reference (RDD) (DFS) (URI) 17
  • 18. A Brief History: Functional Programming for Big Data MapReduce Pregel Giraph Dremel Drill S4 Storm F1 MillWheel General Batch Processing Specialized Systems: Impala GraphLab iterative, interactive, streaming, graph, etc. Tez MR doesn’t compose well for large applications, and so specialized systems emerged as workarounds 18
  • 19. A Brief History: Functional Programming for Big Data circa 2010: a unified engine for enterprise data workflows, based on commodity hardware a decade later… Spark: Cluster Computing with Working Sets Matei Zaharia, Mosharaf Chowdhury, Michael Franklin, Scott Shenker, Ion Stoica people.csail.mit.edu/matei/papers/2010/hotcloud_spark.pdf ! Resilient Distributed Datasets: A Fault-Tolerant Abstraction for In-Memory Cluster Computing Matei Zaharia, Mosharaf Chowdhury, Tathagata Das, Ankur Dave, Justin Ma, Murphy McCauley, Michael Franklin, Scott Shenker, Ion Stoica usenix.org/system/files/conference/nsdi12/nsdi12-final138.pdf 19
  • 20. A Brief History: Functional Programming for Big Data In addition to simple map and reduce operations, Spark supports SQL queries, streaming data, and complex analytics such as machine learning and graph algorithms out-of-the-box. Better yet, combine these capabilities seamlessly into one integrated workflow… 20
  • 21. A Brief History: Key distinctions for Spark vs. MapReduce • generalized patterns ⇒ unified engine for many use cases • lazy evaluation of the lineage graph ⇒ reduces wait states, better pipelining • generational differences in hardware ⇒ off-heap use of large memory spaces • functional programming / ease of use ⇒ reduction in cost to maintain large apps • lower overhead for starting jobs • less expensive shuffles 21
  • 22. TL;DR: Engineering is about costs Sure, maybe you’ll squeeze slightly better performance by using many specialized systems… However, putting on an Eng Director hat, would you be also prepared to pay the corresponding costs of: • learning curves for your developers across several different frameworks • ops for several different kinds of clusters • maintenance + troubleshooting mission-critical apps across several systems • tech-debt for OSS that ignores the math (80 yrs!) plus the fundamental h/w trade-offs 22
  • 24. Spark Deconstructed: Log Mining Example // load error messages from a log into memory! // then interactively search for various patterns! // https://gist.github.com/ceteri/8ae5b9509a08c08a1132! ! // base RDD! val lines = sc.textFile("hdfs://...")! ! // transformed RDDs! val errors = lines.filter(_.startsWith("ERROR"))! val messages = errors.map(_.split("t")).map(r => r(1))! messages.cache()! ! // action 1! messages.filter(_.contains("mysql")).count()! ! // action 2! messages.filter(_.contains("php")).count() 24
  • 25. Driver Worker Worker Worker Spark Deconstructed: Log Mining Example We start with Spark running on a cluster… submitting code to be evaluated on it: 25
  • 26. Spark Deconstructed: Log Mining Example // base RDD! val lines = sc.textFile("hdfs://...")! ! // transformed RDDs! val errors = lines.filter(_.startsWith("ERROR"))! val messages = errors.map(_.split("t")).map(r => r(1))! messages.cache()! ! // action 1! messages.filter(_.contains("mysql")).count()! ! // discussing action 2! the other part messages.filter(_.contains("php")).count() 26
  • 27. Spark Deconstructed: Log Mining Example At this point, take a look at the transformed RDD operator graph: scala> messages.toDebugString! res5: String = ! MappedRDD[4] at map at <console>:16 (3 partitions)! MappedRDD[3] at map at <console>:16 (3 partitions)! FilteredRDD[2] at filter at <console>:14 (3 partitions)! MappedRDD[1] at textFile at <console>:12 (3 partitions)! HadoopRDD[0] at textFile at <console>:12 (3 partitions) 27
  • 28. Driver Worker Worker Worker Spark Deconstructed: Log Mining Example // base RDD! val lines = sc.textFile("hdfs://...")! ! // transformed RDDs! val errors = lines.filter(_.startsWith("ERROR"))! val messages = errors.map(_.split("t")).map(r => r(1))! messages.cache()! ! // action 1! messages.filter(_.contains("mysql")).count()! ! // action 2! medssaigsesc.fuilstesr(i_n.cognt atinhs(e"ph po")t).hcoeuntr() part 28
  • 29. Driver Worker Worker block 1 Worker block 2 block 3 Spark Deconstructed: Log Mining Example // base RDD! val lines = sc.textFile("hdfs://...")! ! // transformed RDDs! val errors = lines.filter(_.startsWith("ERROR"))! val messages = errors.map(_.split("t")).map(r => r(1))! messages.cache()! ! // action 1! messages.filter(_.contains("mysql")).count()! ! // action 2! medssaigsesc.fuilstesr(i_n.cognt atinhs(e"ph po")t).hcoeuntr() part 29
  • 30. Driver Worker Worker block 1 Worker block 2 block 3 Spark Deconstructed: Log Mining Example // base RDD! val lines = sc.textFile("hdfs://...")! ! // transformed RDDs! val errors = lines.filter(_.startsWith("ERROR"))! val messages = errors.map(_.split("t")).map(r => r(1))! messages.cache()! ! // action 1! messages.filter(_.contains("mysql")).count()! ! // action 2! medssaigsesc.fuilstesr(i_n.cognt atinhs(e"ph po")t).hcoeuntr() part 30
  • 31. Driver Worker Worker block 1 Worker block 2 block 3 read HDFS block read HDFS block read HDFS block Spark Deconstructed: Log Mining Example // base RDD! val lines = sc.textFile("hdfs://...")! ! // transformed RDDs! val errors = lines.filter(_.startsWith("ERROR"))! val messages = errors.map(_.split("t")).map(r => r(1))! messages.cache()! ! // action 1! messages.filter(_.contains("mysql")).count()! ! // action 2! medssaigsesc.fuilstesr(i_n.cognt atinhs(e"ph po")t).hcoeuntr() part 31
  • 32. Driver cache 1 Worker Worker block 1 Worker block 2 block 3 cache 2 cache 3 process, cache data process, cache data process, cache data Spark Deconstructed: Log Mining Example // base RDD! val lines = sc.textFile("hdfs://...")! ! // transformed RDDs! val errors = lines.filter(_.startsWith("ERROR"))! val messages = errors.map(_.split("t")).map(r => r(1))! messages.cache()! ! // action 1! messages.filter(_.contains("mysql")).count()! ! // action 2! medssaigsesc.fuilstesr(i_n.cognt atinhs(e"ph po")t).hcoeuntr() part 32
  • 33. Driver cache 1 Worker Worker block 1 Worker block 2 block 3 cache 2 cache 3 Spark Deconstructed: Log Mining Example // base RDD! val lines = sc.textFile("hdfs://...")! ! // transformed RDDs! val errors = lines.filter(_.startsWith("ERROR"))! val messages = errors.map(_.split("t")).map(r => r(1))! messages.cache()! ! // action 1! messages.filter(_.contains("mysql")).count()! ! // action 2! medssaigsesc.fuilstesr(i_n.cognt atinhs(e"ph po")t).hcoeuntr() part 33
  • 34. // base RDD! val lines = sc.textFile("hdfs://...")! ! // transformed RDDs! val errors = lines.filter(_.startsWith("ERROR"))! val messages = errors.map(_.split("t")).map(r => r(1))! messages.cache()! ! // action 1! messages.filter(_.contains("mysql")).count()! ! // action 2! messages.filter(_.contains("php")).count() Driver cache 1 Worker Worker block 1 Worker block 2 block 3 cache 2 cache 3 Spark Deconstructed: Log Mining Example discussing the other part 34
  • 35. Driver cache 1 Worker Worker block 1 Worker block 2 block 3 cache 2 cache 3 process from cache process from cache process from cache Spark Deconstructed: Log Mining Example // base RDD! val lines = sc.textFile("hdfs://...")! ! // discussing transformed RDDs! val errors = lines.filter(_.the startsWith("other ERROR"))part ! val messages = errors.map(_.split("t")).map(r => r(1))! messages.cache()! ! // action 1! messages.filter(_.contains(“mysql")).count()! ! // action 2! messages.filter(_.contains("php")).count() 35
  • 36. Driver cache 1 Worker Worker block 1 Worker block 2 block 3 cache 2 cache 3 Spark Deconstructed: Log Mining Example // base RDD! val lines = sc.textFile("hdfs://...")! ! // discussing transformed RDDs! val errors = lines.filter(_.the startsWith("other ERROR"))part ! val messages = errors.map(_.split("t")).map(r => r(1))! messages.cache()! ! // action 1! messages.filter(_.contains(“mysql")).count()! ! // action 2! messages.filter(_.contains("php")).count() 36
  • 37. Spark Deconstructed: Looking at the RDD transformations and actions from another perspective… action value RDD RDD RDD // load error messages from a log into memory! // then interactively search for various patterns! // https://gist.github.com/ceteri/8ae5b9509a08c08a1132! ! // base RDD! val lines = sc.textFile("hdfs://...")! ! // transformed RDDs! val errors = lines.filter(_.startsWith("ERROR"))! val messages = errors.map(_.split("t")).map(r => r(1))! messages.cache()! ! // action 1! messages.filter(_.contains("mysql")).count()! ! // action 2! messages.filter(_.contains("php")).count() transformations RDD 37
  • 38. Spark Deconstructed: RDD // base RDD! val lines = sc.textFile("hdfs://...") 38
  • 39. RDD RDD RDD Spark Deconstructed: transformations RDD // transformed RDDs! val errors = lines.filter(_.startsWith("ERROR"))! val messages = errors.map(_.split("t")).map(r => r(1))! messages.cache() 39
  • 40. action value RDD RDD RDD Spark Deconstructed: transformations RDD // action 1! messages.filter(_.contains("mysql")).count() 40
  • 42. Unifying the Pieces: Spark SQL // http://spark.apache.org/docs/latest/sql-programming-guide.html! ! val sqlContext = new org.apache.spark.sql.SQLContext(sc)! import sqlContext._! ! // define the schema using a case class! case class Person(name: String, age: Int)! ! // create an RDD of Person objects and register it as a table! val people = sc.textFile("examples/src/main/resources/ people.txt").map(_.split(",")).map(p => Person(p(0), p(1).trim.toInt))! ! people.registerAsTempTable("people")! ! // SQL statements can be run using the SQL methods provided by sqlContext! val teenagers = sql("SELECT name FROM people WHERE age >= 13 AND age <= 19")! ! // results of SQL queries are SchemaRDDs and support all the ! // normal RDD operations…! // columns of a row in the result can be accessed by ordinal! teenagers.map(t => "Name: " + t(0)).collect().foreach(println) 42
  • 43. Unifying the Pieces: Spark Streaming // http://spark.apache.org/docs/latest/streaming-programming-guide.html! ! import org.apache.spark.streaming._! import org.apache.spark.streaming.StreamingContext._! ! // create a StreamingContext with a SparkConf configuration! val ssc = new StreamingContext(sparkConf, Seconds(10))! ! // create a DStream that will connect to serverIP:serverPort! val lines = ssc.socketTextStream(serverIP, serverPort)! ! // split each line into words! val words = lines.flatMap(_.split(" "))! ! // count each word in each batch! val pairs = words.map(word => (word, 1))! val wordCounts = pairs.reduceByKey(_ + _)! ! // print a few of the counts to the console! wordCounts.print()! ! ssc.start() // start the computation! ssc.awaitTermination() // wait for the computation to terminate 43
  • 44. MLI: An API for Distributed Machine Learning Evan Sparks, Ameet Talwalkar, et al. International Conference on Data Mining (2013) http://arxiv.org/abs/1310.5426 Unifying the Pieces: MLlib // http://spark.apache.org/docs/latest/mllib-guide.html! ! val train_data = // RDD of Vector! val model = KMeans.train(train_data, k=10)! ! // evaluate the model! val test_data = // RDD of Vector! test_data.map(t => model.predict(t)).collect().foreach(println)! 44
  • 45. Unifying the Pieces: GraphX // http://spark.apache.org/docs/latest/graphx-programming-guide.html! ! import org.apache.spark.graphx._! import org.apache.spark.rdd.RDD! ! case class Peep(name: String, age: Int)! ! val vertexArray = Array(! (1L, Peep("Kim", 23)), (2L, Peep("Pat", 31)),! (3L, Peep("Chris", 52)), (4L, Peep("Kelly", 39)),! (5L, Peep("Leslie", 45))! )! val edgeArray = Array(! Edge(2L, 1L, 7), Edge(2L, 4L, 2),! Edge(3L, 2L, 4), Edge(3L, 5L, 3),! Edge(4L, 1L, 1), Edge(5L, 3L, 9)! )! ! val vertexRDD: RDD[(Long, Peep)] = sc.parallelize(vertexArray)! val edgeRDD: RDD[Edge[Int]] = sc.parallelize(edgeArray)! val g: Graph[Peep, Int] = Graph(vertexRDD, edgeRDD)! ! val results = g.triplets.filter(t => t.attr > 7)! ! for (triplet <- results.collect) {! println(s"${triplet.srcAttr.name} loves ${triplet.dstAttr.name}")! } 45
  • 46. Demos, as time permits: brand new Python support for Streaming in 1.2 github.com/apache/spark/tree/master/examples/src/main/ python/streaming Twitter Streaming Language Classifier databricks.gitbooks.io/databricks-spark-reference-applications/ content/twitter_classifier/README.html ! ! For more Spark learning resources online: databricks.com/spark-training-resources 46
  • 48. Spark Integrations: Discover Insights Clean Up Your Data Run Sophisticated Analytics Integrate With Many Other Systems Use Lots of Different Data Sources cloud-based notebooks… ETL… the Hadoop ecosystem… widespread use of PyData… advanced analytics in streaming… rich custom search… web apps for data APIs… low-latency + multi-tenancy… 48
  • 49. Spark Integrations: Unified platform for building Big Data pipelines Databricks Cloud databricks.com/blog/2014/07/14/databricks-cloud-making- big-data-easy.html youtube.com/watch?v=dJQ5lV5Tldw#t=883 49
  • 50. Spark Integrations: The proverbial Hadoop ecosystem Spark + Hadoop + HBase + etc. mapr.com/products/apache-spark vision.cloudera.com/apache-spark-in-the-apache-hadoop-ecosystem/ hortonworks.com/hadoop/spark/ databricks.com/blog/2014/05/23/pivotal-hadoop-integrates-the- full-apache-spark-stack.html unified compute hadoop ecosystem 50
  • 51. Spark Integrations: Leverage widespread use of Python Spark + PyData spark-summit.org/2014/talk/A-platform-for-large-scale-neuroscience cwiki.apache.org/confluence/display/SPARK/PySpark+Internals unified compute Py Data 51
  • 52. Spark Integrations: Advanced analytics for streaming use cases Kafka + Spark + Cassandra datastax.com/documentation/datastax_enterprise/4.5/ datastax_enterprise/spark/sparkIntro.html http://helenaedelson.com/?p=991 github.com/datastax/spark-cassandra-connector github.com/dibbhatt/kafka-spark-consumer unified compute data streams columnar key-value 52
  • 53. Spark Integrations: Rich search, immediate insights Spark + ElasticSearch databricks.com/blog/2014/06/27/application-spotlight-elasticsearch. unified compute html elasticsearch.org/guide/en/elasticsearch/hadoop/current/ spark.html spark-summit.org/2014/talk/streamlining-search-indexing- using-elastic-search-and-spark document search 53
  • 54. Spark Integrations: Building data APIs with web apps Spark + Play typesafe.com/blog/apache-spark-and-the-typesafe-reactive- platform-a-match-made-in-heaven unified compute web apps 54
  • 55. Spark Integrations: The case for multi-tenancy Spark + Mesos spark.apache.org/docs/latest/running-on-mesos.html + Mesosphere + Google Cloud Platform ceteri.blogspot.com/2014/09/spark-atop-mesos-on-google-cloud.html unified compute cluster resources 55
  • 57. Spark at Twitter: Evaluation & Lessons Learnt Sriram Krishnan slideshare.net/krishflix/seattle-spark-meetup-spark- at-twitter • Spark can be more interactive, efficient than MR • Support for iterative algorithms and caching • More generic than traditional MapReduce • Why is Spark faster than Hadoop MapReduce? • Fewer I/O synchronization barriers • Less expensive shuffle • More complex the DAG, greater the performance improvement 57 Because Use Cases: Twitter
  • 58. Using Spark to Ignite Data Analytics ebaytechblog.com/2014/05/28/using-spark-to-ignite- data-analytics/ 58 Because Use Cases: eBay/PayPal
  • 59. Hadoop and Spark Join Forces in Yahoo Andy Feng spark-summit.org/talk/feng-hadoop-and-spark-join- forces-at-yahoo/ 59 Because Use Cases: Yahoo!
  • 60. Collaborative Filtering with Spark Chris Johnson slideshare.net/MrChrisJohnson/collaborative-filtering- with-spark • collab filter (ALS) for music recommendation • Hadoop suffers from I/O overhead • show a progression of code rewrites, converting a Hadoop-based app into efficient use of Spark 60 Because Use Cases: Spotify
  • 61. Because Use Cases: Sharethrough Sharethrough Uses Spark Streaming to Optimize Bidding in Real Time Russell Cardullo, Michael Ruggier 2014-03-25 databricks.com/blog/2014/03/25/ sharethrough-and-spark-streaming.html • the profile of a 24 x 7 streaming app is different than an hourly batch job… • take time to validate output against the input… • confirm that supporting objects are being serialized… • the output of your Spark Streaming job is only as reliable as the queue that feeds Spark… • monoids… 61
  • 62. Because Use Cases: Ooyala Productionizing a 24/7 Spark Streaming service on YARN Issac Buenrostro, Arup Malakar 2014-06-30 spark-summit.org/2014/talk/ productionizing-a-247-spark-streaming-service- on-yarn • state-of-the-art ingestion pipeline, processing over two billion video events a day • how do you ensure 24/7 availability and fault tolerance? • what are the best practices for Spark Streaming and its integration with Kafka and YARN? • how do you monitor and instrument the various 62 stages of the pipeline?
  • 63. Because Use Cases: Viadeo Spark Streaming As Near Realtime ETL Djamel Zouaoui 2014-09-18 slideshare.net/DjamelZouaoui/spark-streaming • Spark Streaming is topology-free • workers and receivers are autonomous and independent • paired with Kafka, RabbitMQ • 8 machines / 120 cores • use case for recommender system • issues: how to handle lost data, serialization 63
  • 64. Because Use Cases: Stratio Stratio Streaming: a new approach to Spark Streaming David Morales, Oscar Mendez 2014-06-30 spark-summit.org/2014/talk/stratio-streaming- a-new-approach-to-spark-streaming • Stratio Streaming is the union of a real-time messaging bus with a complex event processing engine using Spark Streaming • allows the creation of streams and queries on the fly • paired with Siddhi CEP engine and Apache Kafka • added global features to the engine such as auditing 64 and statistics
  • 65. Because Use Cases: Guavus Guavus Embeds Apache Spark into its Operational Intelligence Platform Deployed at the World’s Largest Telcos Eric Carr 2014-09-25 databricks.com/blog/2014/09/25/guavus-embeds-apache-spark-into- its-operational-intelligence-platform-deployed-at-the-worlds- largest-telcos.html • 4 of 5 top mobile network operators, 3 of 5 top Internet backbone providers, 80% MSOs in NorAm • analyzing 50% of US mobile data traffic, +2.5 PB/day • latency is critical for resolving operational issues before they cascade: 2.5 MM transactions per second • “analyze first” not “store first ask questions later” 65
  • 66. Why Spark is the Next Top (Compute) Model Dean Wampler slideshare.net/deanwampler/spark-the-next-top- compute-model • Hadoop: most algorithms are much harder to implement in this restrictive map-then-reduce model • Spark: fine-grained “combinators” for composing algorithms • slide #67, any questions? 66 Because Use Cases: Typesafe
  • 67. Installing the Cassandra / Spark OSS Stack Al Tobey tobert.github.io/post/2014-07-15-installing-cassandra- spark-stack.html • install+config for Cassandra and Spark together • spark-cassandra-connector integration • examples show a Spark shell that can access tables in Cassandra as RDDs with types pre-mapped and ready to go 67 Because Use Cases: DataStax
  • 68. One platform for all: real-time, near-real-time, and offline video analytics on Spark Davis Shepherd, Xi Liu spark-summit.org/talk/one-platform-for-all-real- time-near-real-time-and-offline-video-analytics- on-spark 68 Because Use Cases: Conviva
  • 70. Approximations 19-20c. statistics emphasized defensibility in lieu of predictability, based on analytic variance and goodness-of-fit tests ! That approach inherently led toward a manner of computational thinking based on batch windows ! They missed a subtle point… 70
  • 71. 21c. shift towards modeling based on probabilistic approximations: trade bounded errors for greatly reduced resource costs highlyscalable.wordpress.com/2012/05/01/ probabilistic-structures-web-analytics-data- mining/ 71 Approximations
  • 72. 21c. shift towards modeling based on probabil approximations: trade bounded errors for greatly reduced resource costs Twitter catch-phrase: “Hash, don’t sample” highlyscalable.wordpress.com/2012/05/01/ probabilistic-structures-web-analytics-data- mining/ 72 Approximations
  • 73. a fascinating and relatively new area, pioneered by relatively few people – e.g., Philippe Flajolet provides approximation, with error bounds – in general uses significantly less resources (RAM, CPU, etc.) many algorithms can be constructed from combinations of read and write monoids aggregate different ranges by composing hashes, instead of repeating full-queries 73 Approximations
  • 74. algorithm use case example Count-Min Sketch frequency summaries code HyperLogLog set cardinality code Bloom Filter set membership MinHash set similarity DSQ streaming quantiles SkipList ordered sequence search 74 Approximations
  • 75. algorithm use case example Count-Min Sketch frequency summaries code HyperLogLog set cardinality code suggestion: consider these as your most quintessential collections data types at scale Bloom Filter set membership MinHash set similarity DSQ streaming quantiles SkipList ordered sequence search 75 Approximations
  • 76. • sketch algorithms: trade bounded errors for orders of magnitude less required resources, e.g., fit more complex apps in memory • multicore + large memory spaces (off heap) are increasing the resources per node in a cluster • containers allow for finer-grain allocation of cluster resources and multi-tenancy • monoids, etc.: guarantees of associativity within the code allow for more effective distributed computing, e.g., partial aggregates • less resources must be spent sorting/windowing data prior to working with a data set • real-time apps, which don’t have the luxury of anticipating data partitions, can respond quickly 76 Approximations
  • 77. Probabilistic Data Structures for Web Analytics and Data Mining Ilya Katsov (2012-05-01) A collection of links for streaming algorithms and data structures Debasish Ghosh Aggregate Knowledge blog (now Neustar) Timon Karnezos, Matt Curcio, et al. Probabilistic Data Structures and Breaking Down Big Sequence Data C. Titus Brown, O'Reilly (2010-11-10) Algebird Avi Bryant, Oscar Boykin, et al. Twitter (2012) Mining of Massive Datasets Jure Leskovec, Anand Rajaraman, Jeff Ullman, Cambridge (2011) 77 Approximations
  • 79. certification: Apache Spark developer certificate program • http://oreilly.com/go/sparkcert • defined by Spark experts @Databricks • assessed by O’Reilly Media • establishes the bar for Spark expertise 79
  • 80. community: spark.apache.org/community.html video+slide archives: spark-summit.org events worldwide: goo.gl/2YqJZK resources: databricks.com/spark-training-resources workshops: databricks.com/spark-training Intro to Spark Spark AppDev Spark DevOps Spark DataSci Distributed ML on Spark Streaming Apps on Spark Spark + Cassandra 80
  • 81. books: Fast Data Processing with Spark Holden Karau Packt (2013) shop.oreilly.com/product/ 9781782167068.do Spark in Action Chris Fregly Manning (2015*) sparkinaction.com/ Learning Spark Holden Karau, Andy Konwinski, Matei Zaharia O’Reilly (2015*) shop.oreilly.com/product/ 0636920028512.do 81
  • 82. events: Big Data Spain Madrid, Nov 17-18 bigdataspain.org Strata EU Barcelona, Nov 19-21 strataconf.com/strataeu2014 Data Day Texas Austin, Jan 10 datadaytexas.com Strata CA San Jose, Feb 18-20 strataconf.com/strata2015 Spark Summit East NYC, Mar 18-19 spark-summit.org/east Strata EU London, May 5-7 strataconf.com/big-data-conference-uk-2015 Spark Summit 2015 SF, Jun 15-17 spark-summit.org 82
  • 83. presenter: monthly newsletter for updates, events, conf summaries, etc.: liber118.com/pxn/ Just Enough Math O’Reilly, 2014 justenoughmath.com preview: youtu.be/TQ58cWgdCpA Enterprise Data Workflows with Cascading O’Reilly, 2013 shop.oreilly.com/product/ 0636920028536.do 83