Block Sampling: Efficient Accurate Online Aggregation in MapReduce
1 like608 views
The paper discusses block sampling as a method for efficient online aggregation in MapReduce, enabling quicker access to useful results even before job completion. It highlights challenges such as slow random disk access and the need for effective sampling strategies given unstructured data. The authors propose a technique that uses in-memory shuffling to improve sampling rates and accuracy while reducing the communication costs among mapper tasks.
Block Sampling: Efficient Accurate Online Aggregation in MapReduce
- 1. Block Sampling: Efficient Accurate Online Aggregation in MapReduce 5th IEEE International Conference on Cloud Computing Technology and Science (CloudCom 2013) Vasiliki Kalavri, Vaidas Brundza, Vladimir Vlassov {kalavri, vaidas, vladv}@kth.se 3 December 2013, Bristol, UK
- 2. Problem and Motivation Luckily, in many cases results can be useful even before job completion ○ tolerate some inaccuracy ○ benefit from faster answers 2 Big data processing is usually very time- consuming... … but many applications require results really fast or can only use results for a limited window of time
- 3. MapReduce vs. MapReduce Online mapper reducer Local Disk Input Record map function Output Record HTTP request In original MR, a reducer task cannot fetch the output of a map task which hasn't committed its output to disk mapper reducer Input Record map function Output Record TCP- push/pull 3
- 4. Online Aggregation ● Apply the reduce function to the data seen so far ● % input processed to estimate accuracy 4
- 5. Sampling Challenges ● Data in HDFS ○ Disk already access is terribly slow ○ Random disk access for sampling is even slower ● Unstructured Data ○ Sample based on what? ○ We don’t know the query, we don’t know the key or the value! 5
- 6. The Block Sampling Technique 6
- 7. MapReduce Online vs. Block Sampling Average Temperature Estimation on Weather Data Unsorted Sorted 7
- 8. Takeaway 8 ● Useful results even before job completion ● Disk random access is prohibitively expensive → efficiently emulate sampling using in-memory shuffling ● Higher sampling rate improves accuracy but also increases communication costs among mapper tasks
- 9. Block Sampling: Efficient Accurate Online Aggregation in MapReduce 5th IEEE International Conference on Cloud Computing Technology and Science (CloudCom 2013) Vasiliki Kalavri, Vaidas Brundza, Vladimir Vlassov {kalavri, vaidas, vladv}@kth.se 3 December 2013, Bristol, UK
- 10. Average Temperature Estimation on Sorted and Unsorted Weather Data Unsorted Sorted 6 How do the block sampling rate and the % of processed input affect accuracy?
- 11. Performance - Sampling Rate
- 12. Performance - Bias Reduction snapshot freq = 10%
- 13. Experimental Setup ● 8 large-instance OpenStack VMs ○ 4 vCPUs, 8 GB memory, 90 GB disk ● Linux Ubuntu 12.04.2 LTS OSm Java 1.7.0 14 ● up to 17 map tasks and 5 reduce tasks per job, HDFS block size of 64MB ● weather station data from the National Climatic Data Center ftp server (available years 1901 to 2013) ● the complete Project Gutenberg e-books catalog (30615 e-books in .txt format)
- 15. Bias Reduction ● Access Phase: Store the entire input split in the reader task’s local memory ● Shuffling Phase: Shuffle the records of the block in-place ● Processing Phase: Serve a record to the mapper task from local memory (avoids additional disk I/O)
- 16. Future Work ● Integrate statistical estimators ○ provide error bounds for users ● Automatically fine-tune sampling parameters based on system configuration ● Explore alternative sampling techniques and wavelet-approximation