Intro to Spark development

Editor's Notes

• #2: Greet students and mention: I’d like to start by mentioning that this slide deck is being released under a Creative Commons license which allows everyone to download and share this deck as long as you don’t use it commercially. This talk will shortly be on YouTube, so don’t worry about taking notes Mention WIFI details: ??? So, today, I’d like to bootstrap you on your journey of learning and using Spark. By the end of today, you will be familiar with Spark’s architecture and programming API and will have written simple Spark applications. “When you first look at Spark, it can be somewhat intimidating. One primary goal for today is to demystify Spark, so you can step into using it with confidence.”
• #3: Databricks is a company that was founded by the creators of the Spark project in 2013. And one of the first things the founders did after starting the company was donating Spark to the Apache Software Foundation and they remain the largest contributor to Spark’s code base. Databricks Cloud is a end-to-end hosted service. The goal of our product is to be the easiest place to be up and running with Spark with little or no up-front hardware or cost investment, since it’s a cloud service. We want to help our customers quickly pull out insights via visualization from your data… without worrying about the intricacies of installing and configuring a large Hadoop cluster.
• #5: UPDATE This will be a fast-paced, very technical and vendor agnostic class on Spark
• #7: Mention that you’ve taught classes on Python, Ruby, Scala and Java
• #8: Developer: 42 people Data Scientists: 15 people
• #10: IT: 26 people Banking: 8 people
• #11: 52 people have not attended any previous training
• #17: HDFS: 34 people or 58% of audience MapReduce: 24 people or 42% of audience
• #18: 50 people or 86% would like us to focus on Development today
• #19: http://db-engines.com/en/ One way to think about the era of Big Data is as the Storage buildout era and the Compute buildout era. From 2008 – 2013, it was primarily about figuring out whether this NoSQL thing was just hype or if there was something worthwhile to it. The general consensus seems to be that for certain use cases involving very large scale or specialized queries, it does make sense to break out of the relational model and store data in specialized databases. But after moving 10TB of data from Oracle to Cassandra, what do you do with it next? Applying the same point queries and select statements from the relational era doesn’t fulfil the ambitions of the big data promise. What we were lacking 5 years ago is specialized processing engines to compliment our NoSQL stores. But specialized processing systems came at a high cognitive overload cost. Now, big data engineers didn’t just have to understand a handful of NoSQL architectures, but also a handful of processing architectures. This was too much effort… Enter Spark.
• #20: http://db-engines.com/en/
• #21: http://db-engines.com/en/ The DB-Engines Ranking does not measure the number of installations of the systems, or their use within IT systems. DB Popularity uses: # of mentions of the systems on websites Google Trends Frequency of technical discussions on Strack Overflow and DBA StackExchange # of job listings on indeed.com and SImplyHired # of profiles mentioning the technology on LInkedIN Tweets.
• #22: Mention that first there was a general batch system for all big data processing (MapReduce) Then recently came the short period of specialized systems, but they were quite complicated and different from each other to learn Spark is trying to unify the most popular processing paradigms Mention that now 2 engineers who know the Spark API can do what used to take a team of 10 engineers
• #23: Spark: Schedules computational tasks to run in a cluster Let’s you monitor the progress of your big data application Lets you distribute computational tasks across the cluster
• #24: SparkR was introduced just a few days ago with Spark 1.4. This is Spark’s first new language API since PySpark was added in 2012. SparkR is based on Spark’s parallel DataFrame abstraction. Users can create SparkR DataFrames from “local” R data frames, or from any Spark data source such as Hive, HDFS, Parquet or JSON. SparkR DataFrames support all Spark DataFrame operations including aggregation, filtering, grouping, summary statistics, and other analytical functions. They also supports mixing-in SQL queries, and converting query results to and from DataFrames. Because SparkR uses the Spark’s parallel engine underneath, operations take advantage of multiple cores or multiple machines, and can scale to data sizes much larger than standalone R programs. The new DataFrames API was inspired by data frames in R and Pandas in Python. DataFrames integrate with Python, Java, Scala and R and give you state of the art optimization through the Spark SQL Catalyst optimizer. DataFrames are just a distributed collection of data organized into named columns and can be made from tables in Hive, external databases or existing RDDs. SQL: Spark’s module for working with structured data made of rows and columns. Spark SQL used to work against a special type of RDD called a SchemaRDD, but that is now being replaced with DataFrames. Spark SQL reuses the Hive frontend and metastore, which gives you full compatibility with existing Hive data, queries and UDFs. This allows you to run unmodified Hive queries on existing data warehouses. There is also standard connectivity through JDBC or ODBC via a Simba driver. Tableau uses this Simba driver to send queries down to Spark SQL to run at scale. Streaming: makes it easy to build scalable fault-tolerant streaming applications with stateful exactly-once symantics out of the box. Streaming allows you to reuse the same code for batch processing and stream processing. In 2012, Spark Streaming was able to process over 60 million records per second on 100 nodes at sub-second processing latency, which makes it 2 – 4x faster than comparable systems like Apache Storm on Yahoo’s S4. Netflix is one of the big users of Spark Streaming. (60 million / 100 = 600k) Spark Streaming is able to process 100,000-500,000 records/node/sec. This is much faster than Storm and comparable to other Stream processing systems. Sigmoid was able to consume 480,000 records per second per node machines using Kafka as a source. Kafka: Kafka basically acts as a buffer for incoming data. It is a high-throughput distributed messaging system. So Kafka maintains feeds of messages in categories called topics that get pushed or published into Kafka by producers. Then consumers like Spark Streaming can subscribe to topics and consume the feed of published messages. Each node in a Kafka cluster is called a broker. More than 75% of the time we see Kafka being used instead of Flume. Flume: Distributed log collection and aggregation service for moving large amounts of log data from many different sources to a centralized data store. So with Flume, data from external sources like web servers is consumed by a Flume source. When a Flume source receives an event, it stores it into one or more channels. The channels will keep the event until its consumed by a Flume sink. So, when Flume pushes the data into the sink, that’s where the data is buffered until Spark Streaming pulls the data from the sink. MLlib + GraphX: Mllib is Spark’s scalable machine learning library consisting of common algorithms and utilities including classification, regression, clustering, collaborative filtering, dimensionality reduction. MLlib’s datatypes are vectors and matrices and some of the underlying linear algebra operations on them are provided by Breeze and jblas. The major algorithmic components in Mllib are statistics (like max, min, mean, variance, # of non-zeroes, correlations (Pearson’s and Spearman’s correlations), Stratified Sampling, Hypothesis testing, Random Data Generation, Classification & Regression (like linear models, SVMs, logistic regression, linear regression, naïve Bayes, decision trees, random forests, gradient-boosted trees), Collaborative filtering (ALS), Clustering (K-means), Dimensionality reduction (Singular value decomposition/SVD and Principal Component Analysis/PCA), Feature extraction and transformation, optimization (like stochastic gradient descent and limited memory BFGS). Tachyon: memory based distributed storage system that allows data sharing across cluster frameworks like Spark or Hadoop MapReduce. Project has 60 contributors from 20 institutions. Has a Java-like API similar to that of java.io.File class providing InputStream and OutputStream interfaces. Tachyon also implements the Hadoop FileSystem interface, to allow frameworks that can read from Hadoop Input Formats like MapReduce or Spark to read the data. Tachyon has some interesting features for data in tables… like native support for multi-columned data with the option to put only hot columns in memory to save space. BlinkDB: is an approximate query engine for running interactive SQL queries on large volumes of data. It allows users to trade off query accuracy for response time by running queries on data samples and presenting results annotated with error bars. BlinkDB was demoed in 2012 on a 100 node Amazon EC2 cluster answering a range of queries on 17 TBs of data in less than 2 seconds (which is over 200x faster than Hive) with an error of 2 – 10%. To do this, BlinkDB uses an offline sampling module that creates uniform and stratified samples from underlying data. Two of the big users of BlinkDB are Conviva and Facebook.
• #27: YARN: is a resource manager The cool thing about Spark is that it has a unified API for SQL, ML, and Streaming.
• #29: First open sourced around 2010. Currently there’s 700+ total contributors to Spark and 400K lines of code, and 500+ active production deployments. Spark is: The most active Apache project in contributors per month The most active open source project in a functional language - 700+ contributors, 400,00+ lines of code (most of the new code is the libraries, not the core)
• #32: Paper is 6 pages
• #33: Paper is 13 pages
• #35: In the code: Apply functions to results of SQL queries. Run unmodified Hive queries on existing warehouses. Connect through JDBC or ODBC. Special Interest Group for Management of Data This was an invited paper for the Industrial track to be held in Melbourne, Australia in early June 2015
• #36: GraphX unifies ETL, exploratory analysis, and iterative graph computation within a single system. You can view the same data as both graphs and collections, transform and join graphs with RDDs efficiently, and write custom iterative graph algorithms using the Pregel API.
• #37: BlinkDB: is an approximate query engine for running interactive SQL queries on large volumes of data. It allows users to trade off query accuracy for response time by running queries on data samples and presenting results annotated with error bars. BlinkDB was demoed in 2012 on a 100 node Amazon EC2 cluster answering a range of queries on 17 TBs of data in less than 2 seconds (which is over 200x faster than Hive) with an error of 2 – 10%. To do this, BlinkDB uses an offline sampling module that creates uniform and stratified samples from underlying data. Two of the big users of BlinkDB are Conviva and Facebook.
• #38: State of the art Machine Learning algorithms don’t scale easily b/c it is prohibitive to process all the data points. SO, how do you know when you can stop processing more data b/c you are close enough to the final answer?
• #39: State of the art Machine Learning algorithms don’t scale easily b/c it is prohibitive to process all the data points. SO, how do you know when you can stop processing more data b/c you are close enough to the final answer?
• #41: Book has 4 out of 5 stars between 25 reviews at Amazon.com and O’Reilly as of March 2015.
• #43: Today, we are happy to announce Spark Packages (http://spark-packages.org), a community package index to track the growing number of open source packages and libraries that work with Apache Spark. Spark Packages makes it easy for users to find, discuss, rate, and install packages for any version of Spark, and makes it easy for developers to contribute packages. Spark Packages will feature integrations with various data sources, management tools, higher level domain-specific libraries, machine learning algorithms, code samples, and other Spark content. Thanks to the package authors, the initial listing of packages includes scientific computing libraries, a job execution server, a connector for importing Avro data, tools for launching Spark on Google Compute Engine, and many others. We expect this list to grow substantially in 2015, and to help fuel this growth we’re continuing to invest in extension points to Spark such as the Spark SQL data sources API, the Spark streaming Receiver API, and the Spark ML pipeline API. Package authors who submit a listing retain full rights to your code, including your choice of open-source license.
• #44: Organizations from around the world often build dedicated sort machines (specialized software and sometimes specialized hardware) to compete in this benchmark.. Spark actually tied for 1st place with a team from University of California San Diego who have been working on creating a specialized sorting system called TritonSort. Winning this benchmark as a general, fault-tolerant system marks an important milestone for the Spark project. It demonstrates that Spark is fulfilling its promise to serve as a faster and more scalable engine for data processing of all sizes, from GBs to TBs to PBs.  Named after Jim Gray, the benchmark workload is resource intensive by any measure: sorting 100 TB of data following the strict rules generates 500 TB of disk I/O and 200 TB of network I/O. Requires read and write of 500 TB of disk I/O and 200 TB of network (b/c you have to replicate the output to make it fault taulerant) First time a system based on a public cloud system has won
• #46: Zero-Copy (terminology): With Netty, the data from disk only gets sent to one NIC buffer and sent out to other node from there. Older implementation (normal) would have to first copy to FileSystem kernel buffer cache, then to a Executor JVM user-space buffer, and then to kernel NIC buffer and out to the remove reducer node GC: Netty does an explicit managed memory, malloc (outside of JVM). Netty is in the normal Executor JVM process that allocates a bunch of memory buffers off-heap and manages these transport buffers entirely by itself.
• #47: In HDFS, reads normally go through the DataNode. Thus, when the client asks the DataNode to read a file, the DataNode reads that file off of the disk and sends the data to the client over a TCP socket. So-called "short-circuit" reads bypass the DataNode, allowing the client to read the file directly. Obviously, this is only possible in cases where the client is co-located with the data. Short-circuit reads provide a substantial performance boost to many applications. <name>dfs.client.read.shortcircuit</name> <value>true</value>
• #48: Let students know that they can go to Databricks.com to request the expanded 3-day version of this class for their own teams on-site. Introduce Ben/Tag from NewCircle as our training partner company who students can talk to about purchasing Spark classes
• #51: When you start a driver program, it connects to the Spark Cluster You can make the Driver HA in standalone by setting the Boolean for –supervise to true when using Spark Submit (https://github.com/apache/spark/search?utf8=%E2%9C%93&q=supervise) In YARN the driver is HA via YARN primitives
• #52: This RDD has 5 partitions. An RDD is simply a distributed collection of elements. You can think of the distributed collections like of like an array or list in your single machine program, except that it’s spread out across multiple nodes in the cluster. In Spark all work is expressed as either creating new RDDs, transforming existing RDDs, or calling operations on RDDs to compute a result. Under the hood, Spark automatically distributes the data contained in RDDs across your cluster and parallelizes the operations you perform on them. So, Spark gives you APIs and functions that lets you do something on the whole collection in parallel using all the nodes.
• #53: ----- Meeting Notes (6/15/15 15:53) ----- Each data source feeds the partitions differently. e.g., Each HDFS block maps to a partition. In Cassandra, by default, 100,000 rows per partition.
• #56: Introduce that Spark has Operations which can be transformations or actions. Those are 4 green unique blocks in a single HDFS file Here we are filtering out the warnings and info messages so we are left with just errors in the RDD. This doesn’t actually read the file from HDFS just yet… we’re just building out a lineage graph
• #58: directed acyclic graph. That is, it is formed by a collection of vertices and directed edges, each edge connecting one vertex to another, such that there is no way to start at some vertex v and follow a sequence of edges that eventually loops back to v again. A collection of tasks that must be ordered into a sequence, subject to constraints that certain tasks must be performed earlier than others, may be represented as a DAG with a vertex for each task and an edge for each constraint https://en.wikipedia.org/wiki/Directed_acyclic_graph
• #61: ----- Meeting Notes (6/15/15 16:02) ----- This is a stage (which we'll talk about later).
• #62: Now the RDDs dissapear and get destroyed
• #65: It’s okay if only part of the RDD actually fits in memory Talk about lineage: parent RDD and child RDD
• #66: ----- Meeting Notes (6/15/15 16:08) ----- Also note that an application can have many such 1 through 4 procedures.
• #68: Actions force the evaluation of the transformations required for the RDD they are called on, since they are required to actually produce output.
• #70: This is an abstract class…
• #73: ----- Meeting Notes (6/15/15 16:08) ----- From here, go to DBC and the lab. From there, to Adam's deck. Then, back here after Adam's deck and Adam's lab.
• #78: Apache Parquet is a columnar storage format available to any project in the Hadoop ecosystem Simba: Supports all major on-premise and cloud Spark distributions Supports all common data types Maps SQL to Spark SQL Only direct, universal ODBC 3.52 data access solution for Apache Spark
• #81: Although  inspired by data frames in R and Python (Pandas), Spark Dataframes was designed from the ground-up to support modern big data and data science applications.
• #82: With a SQLContext, applications can create DataFrames from an existing RDD, from a Hive table, or from data sources. As an example, the following creates a DataFrame based on the content of a JSON file:
• #83: Once built, DataFrames provide a domain-specific language for distributed data manipulation. Here we include some basic examples of structured data processing using DataFrames:
• #84: The sql function on a SQLContext enables applications to run SQL queries programmatically and returns the result as a DataFrame.
• #85: The 1st method works well when you already know the schema while writing your Spark application. While the 2nd method is more verbose, it allows you to construct DataFrames when the columns and their types are not known until runtime.
• #86: Spark SQL can convert an RDD of Row objects to a DataFrame, inferring the datatypes. Rows are constructed by passing a list of key/value pairs as kwargs to the Row class. The keys of this list define the column names of the table, and the types are inferred by looking at the first row. Since we currently only look at the first row, it is important that there is no missing data in the first row of the RDD. In future versions we plan to more completely infer the schema by looking at more data, similar to the inference that is performed on JSON files.
• #87: Spark SQL can convert an RDD of Row objects to a DataFrame, inferring the datatypes. Rows are constructed by passing a list of key/value pairs as kwargs to the Row class. The keys of this list define the column names of the table, and the types are inferred by looking at the first row. Since we currently only look at the first row, it is important that there is no missing data in the first row of the RDD. In future versions we plan to more completely infer the schema by looking at more data, similar to the inference that is performed on JSON files.
• #88: When a dictionary of kwargs cannot be defined ahead of time (for example, the structure of records is encoded in a string, or a text dataset will be parsed and fields will be projected differently for different users), a DataFrame can be created programmatically with three steps. 1) Create an RDD of tuples or lists from the original RDD; 2) Create the schema represented by a StructType matching the structure of tuples or lists in the RDD created in the step 1. 3) Apply the schema to the RDD via createDataFrame method provided by SQLContext.
• #89: The following example shows how to construct DataFrames in Python. A similar API is available in Scala and Java.
• #90: Once built, DataFrames provide a domain-specific language for distributed data manipulation.  Here is an example of using DataFrames to manipulate the demographic data of a large population of users:
• #92: None of the cores or memory has been given out by this Spark Worker yet
• #93: Technically a misconfiguring: Worker has given all its cores to the running executor, but not all of its memory. It cannot start any more executors, because it’s out of cores. It might as well have given out all its memory. Why did this happen? Because programmer constrained the memory (asking for 512M). If the programmer hadn’t put a memory limit on the request, the worker would’ve been “greedy”, giving all available memory to the one executor. (Only one executor would’ve been able to run, just as with the above setup.)ONLY APPLIES TO STANDALONE RESOURCE MANAGER.
• #94: This UI comes from the driver JVM. The jobs page contains detailed execution information for active and recently completed Spark jobs. One very useful piece of information on this page is the progress of running jobs, stages, and tasks. Within each stage, this page provides several metrics that can be used to better understand physical execution. A common use for this page is to assess the performance of a job. A good first step is to look through the stages which make up a job and see whether there are stages that are particularly slow or vary significantly in response time across multiple runs of the same job. If you have an especially expensive stage, you can click through and better understand what user code the stage is associated with.
• #95: A job breaks down into one or more stages. Don’t worry too much about that right now; it’s an intermediate topic. But, once you get into dealing with stages, there’s a UI that exposes information about them. The first stop for learning about the behavior and performance of a Spark application is Spark’s built-in web UI. This is available at the machine where the driver is running at port 4040 by default. On caveat is that in the case of YARN cluster mode, where the application driver runs inside of the cluster, you should access the UI through the ResourceManager, which proxies requests directly to the driver. Once you’ve narrowed down a stage of interest, the stage page can help isolate performance issues. In data-parallel systems such as Spark, a common source of performance issues is skew, which occurs when a small number of tasks take a very large amount of time and hurt the overall performance. The stage page can help you identify skew by looking at the distribution of different metrics over all tasks. A good starting point is the runtime of the task; do a few tasks take much more time than others? If this is the case, you can dig deeper and see what is causing the tasks to be slow. Do a small number of tasks read or write much more data than others? Are tasks running on certain nodes very slow? These are useful first steps when debugging a job. In addition to looking at task skew, it can be helpful to identify how much time tasks are spending in each of the phases of the task lifecycle: reading, computing, and writing. If tasks spend very little time reading or writing data, but take a long time overall, it can be the case that user code itself is expensive. Some tasks may spend almost all of their time reading data from an external storage system, and will not benefit much from additional optimization in Spark since they are bottlenecked on input read.
• #96: The storage page contains information about persisted (cached) RDDs. An RDD is persisted if someone called persist() on the RDD and it was later computed in some job. In some cases, if many RDDs are cached, older ones will fall out of memory to make space for newer ones. This page will tell you exactly what fraction of each RDD is cached and the quantity of data cached in various storage media (disk, memory, etc). It can be good to scan this page and understand whether important datasets are fitting into memory or not.
• #97: This is the screen you see after clicking on the RDD name on the previous slide. NOTE: This is local mode. You can tell because the executor says “localhost” and both executors are on the same port.
• #98: This section enumerates the set of active properties in the environment of your Spark application. The configuration here represents the “ground truth” of your application’s configuration. It can be helpful if you are debugging which configuration flags are enabled, especially if you are using multiple configuration mechanisms. This page will also enumerate jars and files you’ve added to your application, which can be useful when tracing down issues such as missing dependencies.
• #99: This page enumerates the active executors in the application along with some metrics around the processing and storage on each executor. One valuable use of this page is to confirm that your application has the amount of resources you were expecting. A good “first step” when debugging issues to scan this page, since a misconfiguration resulting in fewer executors than expected can, for obvious reasons, hurt performance. It can also be useful to look for executors with anomalous behaviors, such as a very large ratio of failed to successful tasks. An executor with a high failure rate could indicate a misconfiguration or failure on the physical host in question. Simply removing that host from the cluster can improve performance.
• #100: Production Spark programs can be complex, with long workflows comprised of many different stages. Spark 1.4 adds visual debugging and monitoring utilities to understand the runtime behavior of Spark applications. An application timeline viewer profiles the completion of stages and tasks inside a running program.  A job correlates 1:1 with an action (e.g., a collect(), a count(), etc.). All the visualizations here do is capture the existing metrics that were shown before in Spark 1.3 and prior
• #101: - First running 3 stages in parallel, then a 4th stage with collect - These stages ran in about 15-20 seconds - Why 5 stages? There’s a hidden stage behind the tooltip
• #102: Now we have clicked on a stage The color bars are representing a single task This info is gathered via internal metrics in Spark, exposed via REST Most tasks spend majority of the time doing real processing work (green) Scheduler delay: is driver -> executor shipping of task (mostly network delay) In this stage, since there’s no shuffle write time, we can conclude that it’s a result stage as opposed to a shuffle map stage The x axis is time The y axis just shows what’s running in parallel On node 9, there are at most 2 tasks running in parallel. So there are num_cores = 2 on this exec
• #103:  Spark 1.4 also exposes a visual representation of the underlying computation graph (or “DAG”) that is tied directly to metrics of physical execution. Spark streaming adds visual monitoring over data streams, to continuously track the latency and throughput.  It’s doing a wordcount Little black dots = RDDs Green dot = Cached RDD (if you mouse over it, you can see name or type of RDD) Why are there two stages? Because of the shuffle boundary. Because shuffling goes over the network, it has to force a new stage. Andrew Or designed this DAG visualization
• #104: Looking at Stage 0 here (specific RDD types)
• #106: Each task runs on a slot. Task equates to partition. For each partition, you need a task. A task is an item of work that runs on a slot. A slot is a thread. Analogy: thread pools.
• #109: Mention that in MR, you get parallelism with different process IDs, maps & reducers. There can be an issue with a slot being unused for like 15 - 20seconds though, which a new map is being assigned. This is why Facebook made Corona in 2012. Ask how often TaskTracker heartbeats with the JobTracker? Every 3 seconds.. So there can be at least a 3 second delay when scheduling a job. But in Spark you get parallelism with different threads. In MR the slots are hardcoded as either M or R, but not in Spark - In MR it is recommended that each task run for 30 – 60 seconds, but in Spark a task can be as short as 200 ms “ During the given period, MapReduce took around 66 seconds to refill a slot, while Corona took around 55 seconds (an improvement of approximately 17%). In a benchmark run in our simulation cluster, the slot refill times dropped from 10 seconds with MapReduce to 600 milliseconds with Corona.” “In heavy workloads during our testing, the utilization in the Hadoop MapReduce system topped out at 70%. Corona was able to reach more than 95%.” “The job tracker could not handle its dual responsibilities (1) managing the cluster resources and 2) scheduling all user jobs. ) adequately. At peak load, cluster utilization would drop precipitously due to scheduling overhead.” “Another limitation of the Hadoop MapReduce framework was its pull-based scheduling model. Task trackers provide a heartbeat status to the job tracker in order to get tasks to run. Since the heartbeat is periodic, there is always a pre-defined delay when scheduling tasks for any job. For small jobs this delay was problematic.” “Hadoop MapReduce is also constrained by its static slot-based resource management model. Rather than using a true resource management system, a MapReduce cluster is divided into a fixed number of map and reduce slots based on a static configuration – so slots are wasted anytime the cluster workload does not fit the static configuration. Furthermore, the slot-based model makes it hard for non-MapReduce applications to be scheduled appropriately.” “ Corona introduces a cluster manager whose only purpose is to track the nodes in the cluster and the amount of free resources. A dedicated job tracker is created for each job, and can run either in the same process as the client (for small jobs) or as a separate process in the cluster (for large jobs). One major difference from our previous Hadoop MapReduce implementation is that Corona uses push-based, rather than pull-based, scheduling. After the cluster manager receives resource requests from the job tracker, it pushes the resource grants back to the job tracker. Also, once the job tracker gets resource grants, it creates tasks and then pushes these tasks to the task trackers for running. There is no periodic heartbeat involved in this scheduling, so the scheduling latency is minimized.” Corona: https://www.facebook.com/notes/facebook-engineering/under-the-hood-scheduling-mapreduce-jobs-more-efficiently-with-corona/10151142560538920
• #110: Static vs. Dynamic doesn’t refer to RDD partitioning here… In cluster mode Spark depends on a cluster manager to launch executors and, in certain cases, to launch the driver. The cluster manager is a pluggable component in Spark. Each one of these resource managers has different pros and cons. One nice thing about running Spark in YARN, for example, is that you can dynamically resize the # of Executors in the application. This feature is not yet possible in Standalone mode. Static Partitioning: With this approach, each application is given a maximum amount of resources it can use, and holds onto them for its whole duration. Mesos: In “fine-grained” mode (default), each Spark task runs as a separate Mesos task. This allows multiple instances of Spark (and other frameworks) to share machines at a very fine granularity, where each application gets more or fewer machines as it ramps up and down, but it comes with an additional overhead in launching each task. This mode may be inappropriate for low-latency requirements like interactive queries or serving web requests. In Standalone and Mesos course-grained, you can control the maximum number of resources Spark will acquire. By default, it will acquire all cores in the cluster (that get offered by Mesos), which only makes sense if you run just one application at a time. You can cap the max # of cores using conf.set("spark.cores.max", "10") (for example). Standalone mode: By default, applications submitted to the standalone mode cluster will run in FIFO (first-in-first-out) order, and each application will try to use all available nodes. You can limit the number of nodes an application uses by setting the spark.cores.maxconfiguration property in it, or change the default for applications that don’t set this setting through spark.deploy.defaultCores. Finally, in addition to controlling cores, each application’s spark.executor.memory setting controls its memory use. YARN: The --num-executors option to the Spark YARN client controls how many executors it will allocate on the cluster, while --executor-memory and --executor-cores control the resources per executor. The “coarse-grained” mode will instead launch only one long-running Spark task on each Mesos machine, and dynamically schedule its own “mini-tasks” within it. The benefit is much lower startup overhead, but at the cost of reserving the Mesos resources for the complete duration of the application
• #112: Local – run spark with one worker thread Local[N] – run spark with N worker threads . Again, think thread pool. Local[*] – run Spark with as many worker threads as logical cores on your machine Note, that in local mode a shuffle is not costly! Not good for real performance testing. Spark can efficiently support tasks as short as 200 ms - - - - There are a lot of internal threads (I think shuffle servers alone have 8 threads?), but most of them are just idle and don't do much so I wouldn't worry about the overhead. You can launch a spark shell on a single node and then run jstack to see what all the threads are doing.  - - -Ask Reynold: In local mode, can students still change the % of memory allocated to tasks vs shuffle vs user code? A) Yes, same thing.
• #114: Mention how much RAM to give each JVM Mention that OOMs can happen in the driver or executor There will be 3 masters and the workers… when they start up, they ask ZK where the active master is. How to separate RDD caches on disk from shuffle data? A) It is not possible right now Spark_local_dirs: (SSDs) Directory to use for "scratch" space in Spark, including map output files and RDDs that get stored on disk. This should be on a fast, local disk in your system. It can also be a comma-separated list of multiple directories on different disks. NOTE: In Spark 1.0 and later this will be overriden by SPARK_LOCAL_DIRS (Standalone, Mesos) or LOCAL_DIRS (YARN) environment variables set by the cluster manager - - On the 2nd machine, we have SPARK_WORKER_CORES set to 10, instead of 6 like all the other machines. Note, there’s no way to tell a worker that has SPARK_WORKER_CORES set to 15 to just launch Executors with only 10 cores. However, you can start an entire Spark application and say that you only want ‘x’ cores and then some workers will launch Executors, but other workers will not. This could lead to a bit of unbalanced cluster. - - - - ZooKeeper: Utilizing ZooKeeper to provide leader election and some state storage, you can launch multiple Masters in your cluster connected to the same ZooKeeper instance. One will be elected “leader” and the others will remain in standby mode. If the current leader dies, another Master will be elected, recover the old Master’s state, and then resume scheduling. The entire recovery process (from the time the first leader goes down) should take between 1 and 2 minutes. Note that this delay only affects scheduling new applications – applications that were already running during Master failover are unaffected. After you have a ZooKeeper cluster set up, enabling high availability is straightforward. Simply start multiple Master processes on different nodes with the same ZooKeeper configuration (ZooKeeper URL and directory). Masters can be added and removed at any time. In order to schedule new applications or add Workers to the cluster, they need to know the IP address of the current leader. This can be accomplished by simply passing in a list of Masters where you used to pass in a single one. For example, you might start your SparkContext pointing to spark://host1:port1,host2:port2. This would cause your SparkContext to try registering with both Masters – if host1 goes down, this configuration would still be correct as we’d find the new leader, host2. There’s an important distinction to be made between “registering with a Master” and normal operation. When starting up, an application or Worker needs to be able to find and register with the current lead Master. Once it successfully registers, though, it is “in the system” (i.e., stored in ZooKeeper). If failover occurs, the new leader will contact all previously registered applications and Workers to inform them of the change in leadership, so they need not even have known of the existence of the new Master at startup. Due to this property, new Masters can be created at any time, and the only thing you need to worry about is that new applications and Workers can find it to register with in case it becomes the leader. Once registered, you’re taken care of.
• #121: Store RDD as deserialized Java objects in the JVM. If the RDD does not fit in memory, some partitions will not be cached and will be recomputed on the fly each time they're needed. (default)
• #123: Store RDD as serialized Java objects (one byte array per partition). This is generally more space-efficient than deserialized objects Remember to chose a fast serialization library if choosing this! Caching serialized objects will slightly slow down the cache operation due to the cost of serializing objects, but it can substantially reduce time spent on garbage collection in the JVM, since many individual records can be stored as a single serialized buffer. This is because the cost of garbage collection scales with the number of objects on the heap, not the number of bytes of data, and this caching method will take many objects and serialize them into a single giant buffer. Consider this option if you are caching large amounts of data (e.g. Gigabyes) as objects in a single executor and/or seeing long garbage collection pauses. Such pauses would be visible in the application UI under the “GC Time” column for each task.
• #124: Store RDD as deserialized Java objects in the JVM. If the RDD does not fit in memory, store the partitions that don't fit on disk, and read them from there when they're needed. Note that just part of a partition cannot spill to disk. The entire partition will spill if it needs to. To-Do: When it goes down to disk though, it’s always serialized, right?
• #125: Similar to MEMORY_ONLY_SER, but spill partitions that don't fit in memory to disk instead of recomputing them on the fly each time they're needed.
• #126: Similar to MEMORY_ONLY_SER, but spill partitions that don't fit in memory to disk instead of recomputing them on the fly each time they're needed.
• #127: Store RDD as deserialized Java objects in the JVM. If the RDD does not fit in memory, some partitions will not be cached and will be recomputed on the fly each time they're needed. (default)
• #128: Store RDD as deserialized Java objects in the JVM. If the RDD does not fit in memory, store the partitions that don't fit on disk, and read them from there when they're needed. Note that just part of a partition cannot spill to disk. The entire partition will spill if it needs to. To-Do: When it goes down to disk though, it’s always serialized, right?
• #129: To-Do: make a node machine all around to show node-level locality Store RDD in serialized format in Tachyon. Compared to MEMORY_ONLY_SER, OFF_HEAP reduces garbage collection overhead and allows executors to be smaller and to share a pool of memory, making it attractive in environments with large heaps or multiple concurrent applications. Furthermore, as the RDDs reside in Tachyon, the crash of an executor does not lead to losing the in-memory cache. In this mode, the memory in Tachyon is discardable. Thus, Tachyon does not attempt to reconstruct a block that it evicts from memory.
• #131: Spark automatically monitors cache usage on each node and drops out old data partitions in a least-recently-used (LRU) fashion. If you would like to manually remove an RDD instead of waiting for it to fall out of the cache, use the RDD.unpersist() method.
• #132: Spark also automatically persists some intermediate data in shuffle operations (e.g. reduceByKey), even without users calling persist. This is done to avoid recomputing the entire input if a node fails during the shuffle. We still recommend users call persist on the resulting RDD if they plan to reuse it.
• #135: When Spark is transferring data over the network or spilling data to disk, it needs to serialize objects into a binary format. This comes into play during shuffle operations, where potentially large amounts of data are transferred. By default Spark will use Java’s built-in serializer.
• #137: The only reason Kryo is not the default is because of the custom registration requirement, but we recommend trying it in any network-intensive application. Spark automatically includes Kryo serializers for the many commonly-used core Scala classes covered in the AllScalaRegistrar from the Twitter chill library.
• #138: One way to avoid shuffles when joining two datasets is to take advantage of broadcast variables. When one of the datasets is small enough to fit in memory in a single executor, it can be loaded into a hash table on the driver and then broadcast to every executor. A map transformation can then reference the hash table to do lookups.
• #139: When we pass functions to Spark, like map or filter, they can use variables defined outside them in the driver program, but each task running on the cluster gets a new copy of each variable, and updates from these copies are not propagated back to the driver. Recall that Spark automatically sends all variables referenced in your closures to the worker nodes. While this is convenient, it can also be inefficient because (1) the default task launching mechanism is optimized for small task sizes, and (2) you might, in fact, use the same variable in multiple parallel operations, but Spark will send it separately for each operation.
• #141: One way to avoid shuffles when joining two datasets is to take advantage of broadcast variables. When one of the datasets is small enough to fit in memory in a single executor, it can be loaded into a hash table on the driver and then broadcast to every executor. A map transformation can then reference the hash table to do lookups.
• #148: Spark Streaming can handing gigabytes of data per second Note that Python API was added starting Spark 1.2 (it supports all DSTream transformations and almost all output operations). But it currently only supports basic incoming sources like text files and text data over network sockets (Flume + Kafka support in PySpark coming soon)
• #149: Spark Streaming is an extension of the core Spark API that enables scalable, high-throughput, fault-tolerant stream processing of live data streams. Data can be ingested from many sources like Kafka, Flume, Twitter, ZeroMQ, Kinesis or TCP sockets can be processed using complex algorithms expressed with high-level functions like map,reduce, join and window. Finally, processed data can be pushed out to filesystems, databases, and live dashboards. In fact, you can apply Spark’s machine learning and graph processing algorithms on data streams.
• #151: - Streaming Started as research project in 2012 where they had to tear out the internals of the existing version of Spark and had to re-write it to make it 10 times faster than what it was back then.
• #153: An electric company can join 100s of thousands of live data points from the electric grid with live weather data and apply predictive machine learning algorithms to try to predict that a snow storm might be starting in Denver and to start devoting personnel and resources there to fix grid problems.
• #154: Spark Streaming receives live input data streams and divides the data into batches, which are then processed by the Spark engine to generate the final stream of results in batches.
• #156: TO-DO: animate the second one after first. Spark is built on the idea of RDDs. Likewise, Spark Streaming is build on the idea of Dstreams. A DStream is a sequence of data arriving over time. Internally, each DStream is represented as a sequence of RDDs arriving at each time step. Each RDD in a DStream contains data from a certain interval, as shown in the following figure.
• #157: Ask TD: Is that first conversion from Blocks to Partitions called a transformation? Any operation applied on a DStream translates to operations on the underlying RDDs. These underlying RDD transformations are computed by the Spark engine. The DStream operations hide most of these details and provide the developer with higher-level API for convenience. 
• #158: In code, it’s all Stream types… but in the background… in execution layer, it is making RDDs. First, we import StreamingContext, which is the main entry point for all streaming functionality. We create a local StreamingContext with two execution threads, and batch interval of 5 seconds. Using this context, we can create a DStream that represents streaming data from a TCP source, specified as hostname (e.g. localhost) and port (e.g. 9999). This lines DStream represents the stream of data that will be received from the data server. Each record in this DStream is a line of text. Next, we want to split the lines by space into words. flatMap is a one-to-many DStream operation that creates a new DStream by generating multiple new records from each record in the source DStream. In this case, each line will be split into multiple words and the stream of words is represented as the words DStream. Next, we want to count these words. The words DStream is further mapped (one-to-one transformation) to a DStream of (word, 1) pairs, which is then reduced to get the frequency of words in each batch of data. Finally, wordCounts.pprint() will print a few of the counts generated every second. Start receiving data and processing it using streamingContext.start() Wait for the processing to be stopped (manually or due to any error) usingstreamingContext.awaitTermination() The processing can be manually stopped using streamingContext.stop().
• #160: Most input Dstream (except file stream or 1.3 Kafka direct stream) are associated with a Receiver object which receives the data from a source Note that, unlike RDDs, the default persistence level of DStreams keeps the data serialized in memory. From the block, the partition can spill to disk if needed The 2nd Executor for replication is chosen randomly
• #163: In addition to using a garbage collector less likely to introduce pauses, you can make a big difference by reducing GC pressure. Caching RDDs in serialized form (instead of as native objects) also reduces GC pressure, which is why, by default, RDDs generated by Spark Streaming are stored in serialized form. Using Kryo serialization further re‐ duces the memory required for the in memory representation of cached data.
• #164: To-do: is it correct to color the SSD disks from purple to blue? From the block, the partition can spill to disk if needed The 2nd Executor for replication is chosen randomly Show spilling on window operations.. Not really here.
• #165: The Streaming UI exposes statistics for our batch processing and our receivers. In our example we have one network receiver, and we can see the message processing rates. If we were falling behind, we could see how many records each receiver is able to process. We can also see whether a receiver failed. The batch processing statistics show us how long our batches take and also break out the delay in scheduling the job. If a cluster experiences contention then the scheduling delay may increase. The most common question is what minimum batch size Spark Streaming can use. In general, 500 milliseconds has proven to be a good minimum size for many applications. The best approach is to start with a larger batch size (around 10 seconds) and work your way down to a smaller batch size. If the processing times reported in the Streaming UI remain consistent, then you can continue to decrease the batch size, but if they are increasing you may have reached the limit for your application.
• #166: These show DStream operations now like updateStateByKey This also shows the batch time (# after @) batch times can be milliseconds, if they’re running more often than, say, every second 408 is a batch, 409 is different batch (that happened before, stage name is kinda inverted in Streaming)
• #167: Note that Spark worker/executor as a long-running task, hence it occupies one of the cores allocated to the Spark Streaming application. Hence, it is important to remember that Spark Streaming application needs to be allocated enough cores (or threads, if running locally) to process the received data, as well as, to run the receiver(s). When running a Spark Streaming program locally, do not use “local” or “local[1]” as the master URL. Either of these means that only one thread will be used for running tasks locally. If you are using a input DStream based on a receiver (e.g. sockets, Kafka, Flume, etc.), then the single thread will be used to run the receiver, leaving no thread for processing the received data.
• #174: UpdateStateByKey: used to keep track of state across batches. Like if you want to know how many errors came in throughout time. Transform: Return a new DStream by applying a RDD-to-RDD function to every RDD of the source DStream. This can be used to do arbitrary RDD operations on the DStream