MapReduce has recently emerged as a new paradigm for large-scale data analysis due to
its high scalability, fine-grained fault tolerance and easy programming model. Since different
jobs often share similar work (eg, several jobs scan the same input file or produce the same
map output), there are many opportunities to optimize the performance for a batch of jobs. In
this paper, we propose two new techniques for multi-job optimization in the MapReduce
framework. The first is a generalized grouping technique (which generalizes the recently …