Learning spark ch06 - Advanced Spark Programming
- 1. C H A P T E R 0 6 : A D V A N C E D S P A R K P R O G R A M M I N G Learning Spark by Holden Karau et. al.
- 2. Overview: Advanced Spark Programming Introduction Accumulators Accumulators and Fault Tolerance Custom Accumulators Broadcast Variables Optimizing Broadcasts Working on a Per-Partition Basis Piping to External Programs Numeric RDD Operations Conclusion
- 3. 6.1 Introduction This chapter introduces a variety of advanced Spark programming features that we didn’t get to cover in the previous chapters. We introduce two types of shared variables: Accumulators to aggregate information Broadcast variables to efficiently distribute large values. Throughout this chapter we build an example using ham radio operators’ call logs as the input. This chapter introduces how to use Spark’s language- agnostic pipe() method to interact with other programs through standard input and output.
- 4. 6.2 Accumulators When we normally pass functions to Spark, such as a map() function or a condition for filter(), they can use variables defined outside them in the driver program, but each task running on the cluster gets a new copy of each variable, and updates from these copies are not propagated back to the driver. Spark’s shared variables, accumulators and broadcast variables, relax this restriction for two common types of communication patterns: aggregation of results and broadcasts. Spark supports accumulators of type Double, Long, and Float
- 5. Edx and Coursera Courses Introduction to Big Data with Apache Spark Spark Fundamentals I Functional Programming Principles in Scala
- 6. 6.3 Broadcast Variables Spark’s second type of shared variable, broadcast variables, allows the program to efficiently send a large, read-only value to all the worker nodes for use in one or more Spark operations. They come in handy, for example, if your application needs to send a large, read- only lookup table to all the nodes, or even a large feature vector in a machine learning algorithm. When we are broadcasting large values, it is important to choose a data serialization format that is both fast and compact, because the time to send the value over the network can quickly become a bottleneck if it takes a long time to either serialize a value or to send the serialized value over the network.
- 7. 6.4 Working on a Per-Partition Basis Working with data on a per-partition basis allows us to avoid redoing setup work for each data item. Operations like opening a database connection or creating a random-number generator are examples of setup steps that we wish to avoid doing for each element. Spark has per-partition versions of map and for each to help reduce the cost of these operations by letting you run code only once for each partition of an RDD.
- 8. 6.5 Piping to External Programs With three language bindings to choose from out of the box, you may have all the options you need for writing Spark applications. However, if none of Scala, Java, or Python does what you need, then Spark provides a general mechanism to pipe data to programs in other languages, like R scripts. Spark provides a pipe() method on RDDs. Spark’s pipe() lets us write parts of jobs using any language we want as long as it can read and write to Unix standard streams. With pipe(), you can write a transformation of an RDD that reads each RDD element from standard input as a String, manipulates that String however you like, and then writes the result(s) as Strings to standard output.
- 9. 6.6 Numeric RDD Operations Spark’s numeric operations are implemented with a streaming algorithm that allows for building up our model one element at a time. The descriptive statistics are all computed in a single pass over the data and returned as a StatsCounter object by calling stats(). If you want to compute only one of these statistics, you can also call the correspond‐ing method directly on an RDD—for example, rdd.mean() or rdd.sum().
- 10. 6.7 Conclusion In this chapter, you have been introduced to some of the more advanced Spark programming features that you can use to make your programs more efficient or expressive. Subsequent chapters cover deploying and tuning Spark applications, as well as built-in libraries for SQL and streaming and machine learning. We’ll also start seeing more complex and more complete sample applications that make use of much of the functionality described so far, and that should help guide and inspire your own usage of Spark.
- 11. Edx and Coursera Courses Introduction to Big Data with Apache Spark Spark Fundamentals I Functional Programming Principles in Scala