Optimal Strategies for Large-Scale Batch ETL Jobs

Editor's Notes

• #2: Today I’m going to share with you some tips and tricks to help you get started with processing large data in batch processes.
• #3: First, let me tell you a little bit about Neustar. Neustar provides a range of cutting edge marketing solutions, which you can check out online at the above link.
• #4: On our team, we focus on helping the world’s most valuable brands
• #5: A quick word here on our stack, Our jobs run on AWS EMR, and read and output data to S3. we’ve built infrastructure to support our data pipeline, so that we have a fast, highly fault tolerant, cloud based system. All of our Spark jobs shown in the middle blue box are batch ETL jobs. Which is our focus today.
• #6: Our focus today is on Batch ETL jobs. Which differentiates itself from other use cases of Spark such as ad hoc data science uses, or streaming in the following ways. Runs on schedule/ programmatically triggered. So it needs to be reliable and robust, humans will not be manually monitoring the job, or be able to tweak it during runs. Secondly, we Aim for complete utilization of cluster resources, esp. memory and CPU. In streaming, depending on the workload, you might not care as much about using all of your machine resources at every moment. But in batch, the goals is to squeeze every bit of juice out of our machines, so we can arrive at our result with minimal cost.
• #7: We’re getting late arriving data all the time, hence historical state.
• #8: We’re going to try touch upon everything I promised in the talk description. And if we run out of time, the slides will be available online soon. By skew we mean when data is distributed in a way where outliers really affect performance.
• #9: Let’s use a specific problem to get started. This example is relevant to our business, and we will learn about spark in the process. At Neustar, we process large quantities of ad events. Some examples include impressions, clicks, conversions. The problem we’re going to solve is, which impression or click events contributed to the conversion happening? A user usually has many events, so multiple conversions and multiple target events. We need to correctly attributed each of the conversions events for each user. Let’s use a concrete example:
• #10: Let’s use a concrete example. How would we solve this in code?
• #11: Attribute correctly based on timestamp, the target event has been occur before the conversion. We care about the latest target events. Also based on metadata to ensure they are for the same advertiser etc.
• #12: For 90 days of events: we are loading around 100 T of data. For the maximum join, we are joining 50 billion conversions with 250 billion impressions
• #13: Let’s talk about issues you will encounter only at scale
• #14: When working at this scale, you will see errors that other people won’t see. For example, driver OOM. that stacktrace is not actually that JVM is out of memory, your metrics will say you have plenty of heap left. It’s java's ability to allocate an array, max size of array is 2B. we have more than 2G of map status outputs. The byte array out stream cannot grow any more. You can see that it Overflowed the buffer in the ByteArrayOutputStream. What is contained in this output stream?
• #15: We are serializing statuses, which is an array of MapStatuses. The size of the array is the number of map tasks. Each status contains information about how the map block is used by each reducer. We can already see, this is a m times n problem.
• #16: Spark has 2 types of mapStatues: If the number of partitions exceeds 2000, the HighlyCompressedMapStatus is used. If the data also happens to be distributed in a way that would prevent the bitmap from being highly compressible, for example, when each map job output goes to a random set of reducers, with some reducers getting nothing, and some getting output, you can create a situation where your buffer will not be big enough for the compressed statuses! Easiest solution is to reduce partitions #
• #17: Luckily the solution is also transparent.
• #18: For Batch jobs we don’t have a long running application. Having long GC cycles is a waste of resources on our clusters, so we tweak our jobs and cluster settings so we can avoid large amounts of GC. We’re working with large data, which means longer GC when it happens. The setting to tweak here is the spark.cleaner.periodicGC.interval. The cleaning thread will block on cleanup tasks, for example when the driver performs a GC and cleans up all broadcasts. For larger jobs, we increase the periodic GC on the driver so that it is effectively disabled. For this job we set it to 600 minutes Java’s newer G1 GC completely changes the traditional approach. The heap is partitioned into a set of equal-sized heap regions, each a contiguous range of virtual memory (Figure 2). Certain region sets are assigned the same roles (Eden, survivor, old) as in the older collectors, but there is not a fixed size for them. minor GC occurs, G1 copies live objects from one or more regions of the heap to a single region on the heap Full GC occurs only when all regions hold live objects and no full-empty region can be found. greatly improves heap occupancy rate when full GC is triggered, but also makes the minor GC pause times more controllable
• #19: G1GC – low latency high throughput.
• #20: we need Spark to be a little bit more patient when dealing with larger jobs, so we ask that it waits longer for communication between machines in case one of the GC, because our heap size is so large, we will exceeds the timeout.
• #21: Protection against spurious failures which can occur. Spark default maxFailures per task is 3, not enough in large jobs. On the other hand, we don’t want to set it too high, or else, true errors will get retried forever, preventing timely response, and is costly.
• #22: We’re going to try touch upon everything I promised in the talk description. And if we run out of time, the slides will be available online soon.
• #23: At this scale, you’re more likely than not to encounter skew. We had extreme skew in our data.
• #24: Overwhelming majority of our users have fewer than 1000 events. In fact, most have fewer than 50 events.
• #25: Upon closer inspection, our data had very few extreme outliers, out of 20B, there were only 879 that were greater than 75k these very few bad partitions were causing our executors to die
• #27: Executors weren’t as likely to die anymore by grouping in lists.
• #28: We increased partitions and nested our data, and our executors weren’t dying anymore! But there were still inefficiencies caused by skew. Notice the lag in CPU use in ganglia. All of those resources were wasted because there were a few bad partitions.
• #29: The max was taking 50 minutes vs the median of 24 seconds! We need to do something about this.
• #30: If you have domain specific knowledge of your data, use it to filter “bad” data out as early as you can We couldn’t do this in the naïve way because we had no idea which users were going to be bad Salt your data, and join twice. In our empirical studies, this was never worth it since shuffle is so expensive. Use bloom filter if one side of your join is much smaller than the other Bloom filters quickly become large after certain size. For example, if we had xxx conversions, the bloom filter for a 0.1% false positive rate would have been xxx . To broad cast this out and filter would have used more resources than to directly join ----- Meeting Notes (9/25/17 14:38) ----- If
• #31: The size mainly determined by number of items in the filter, and the probability of false positives. The more items you have the bigger the filter. The more accurate the filter is, the bigger the filter The filter can be broadcast out to executors, and used in tasks. uses a bittorrest algo broadcast into slices (couple hundred pieces). overlay a couple of gigabytes. bloom filter: hash functions, more hash functions you can have accuacy. bitset. hash(x) && hash2(x) && hash3(x)
• #32: One thing to note is, Don’t be afraid of using a high false positive rate. If our goal is to shrink the joinable set down, using a 5% false positive rate results in 80% of the data filtered out. This makes subsequent joins much faster
• #33: However, we’ve had great success in many other jobs where the join was even more lopsided. Please experiment and see if it could work for you.
• #34: Now going back to our original problem. You can find bloom filter size calculators online to find out if Bloom filter is a good solution for you. Unfortunately, both side of our join were sizable, and the bloom filter solution was not the right solution for this job.
• #35: None of these strategies worked directly for us, so we were still left with skew.
• #36: Let’s look at the ugly long tail graph again.
• #37: Let’s go back to our problem. We have a very long tail. It is always a good idea to understand what your threads are spending time on, and for long tail, we are especially interested. On our r3.4xls with 16 cores, we saw that most threads were stuck on writing. it was spending a lot of time writing to shuffle space, so let’s try to reduce the size to be written
• #38: On our r3.4xls with 16 cores, we saw that most threads were stuck on writing. it was spending a lot of time writing to shuffle space, so let’s try to reduce the size to be written
• #39: Let’s go back to the data. Inspecting the event data revealed to us that in the long tail, the events had almost identical information, spread over time. If we retain just 1 event per hour, at 90 days, it is at most 2160 events per user. However, this seems to mean that we need to group by userId first, which requires a shuffle, which defeats the whole purpose of this exercise, right? This is where map side combine comes in.
• #40: Let’s go back to the data. Inspecting the event data revealed to us that in the long tail, the events had almost identical information, spread over time. If we retain just 1 event per hour, at 90 days, it is at most 2160 events per user. However, this seems to mean that we need to group by userId first, which requires a shuffle, which defeats the whole purpose of this exercise, right? This is where map side combine comes in. If we use CombineByKey, we can specify the operation to perform on the map side. Here we can thin out the collection.
• #41: The basic structure is shown here. Notice in the second lambda, we rate limit/ thin out collection when list size reaches a certain max. Feel free to check out the slides, they will be available online.
• #42: We got rid of the long tail, but my job is still generally slow, can I squeeze more performance out of it by tweaking my job? The answer is yes.
• #43: We’re going to try touch upon everything I promised in the talk description. And if we run out of time, the slides will be available online soon.
• #44: Reusing the partitioner allows all the data to be partitioned the same way, and ready for joins/reduceByKeys. This way we can transform multiple wide transformations into a single narrow transformation
• #45: Please don’t try to read this, what I want to point out here, is this long narrow stage. It contains 3 combineByKeys, and 2 left outer joins in the same narrow stage. If we had used different partitioners, each would induce a shuffle.
• #46: But we can go further than that to reduce the amount of shuffles. We are willing to pay for it in memory.
• #47: Let’s say we want get count of users by campaign, and we also want a count of users by site. The straightforward way to do this is to take the RDD, cache it, and reduceByKey on it twice. But if we duplicate the rows like this, we are able to reduceByKey once and then filter for the two desired results.
• #48: Another strategy that greatly improves performance is coalescing # of partitions down.
• #49: A simplistic example would be this. We load data from S3, calculate the original partition number, use an algorithm that makes sense for your data to find the new partition #, then coalesce your data down to new # of partitions. Subsequent operations will be more efficient
• #50: Do a count, or a reduce.
• #51: spark is lazy, that you have to materialize first. or else spark will load from source again.
• #52: We’re going to try touch upon everything I promised in the talk description. And if we run out of time, the slides will be available online soon.
• #53: Let’s go through two ganglia use cases. It’s available in AWS EMR.
• #54: What happens when you see waves occurring within a single stage? Sometimes the waves to longer, and takes longer than 10 minutes from crest to trough.
• #55: DISK ONLY caching, it does depend on the machine class. r4 it's not that great. on board SSD is good, EBS is not that great. Tune and test.
• #56: Notice the last stage is only using 70% of CPU for the entire process.
• #57: Decrease number partitions of RDD used in this stage. Less total overhead, fewer tasks means each task computes more data, and this helps with CPU utilization
• #58: After tweaking the above 2 properties, we get much better cluster usage!
• #59: We’ve gone though most of these configurations today, the ones I didn’t mention Spark does a good job notifying users to set when needed. Feel free to reach out to me with any questions about this after this session. spark.dynamicAllocation.enabled FALSE not multi tenented spark.driver.maxResultSize 8g set to really high, since our driver is large. We can handle lots of results going to the driver spark.rpc.message.maxSize 2047 if you have many map and reduce tasks, use max. used to send out mapStatues, which we know is m x n Overhead because JVM overhead. Protect against Executor OOM. what goes into the memory overhead? Intern pool, everything off the JVM. Thread stacks, NIO buffers. Shared native libraries. Yarn coordination process takes 896MB (you can find it on yarn application page)
• #60: Why not dataframes? Complicated logic difficult to express in SQL. For example, finding the latest events that fit specific criteria after a join, or rate limiting data by hour boundaries. Partitions, partitioners, combine by key. Customization of code. Really large data requires specific partition settings, map side combine optimizations, bloom filters in each type of stage. More difficult to do with dataframes. dataframes are great, but RDD gives us flexibility to exploit patterns exist in the data. In R&O we were able to go from down mutiple reduces with data skew. to 1 single pass with RDDS. DF is great to general data analysis tool, RDDs fantastic for power users. 6x difference in size between DF vs RDDs.