Improving the Load Balance of MapReduce Operations based on the Key Distribution of Pairs

Load balance is important for MapReduce to reduce job duration, increase parallel efficiency, etc. Previous work focuses on coarse-grained scheduling. This study concerns fine-grained scheduling on MapReduce operations. Each operation represents one invocation of the Map or Reduce function. Scheduling MapReduce operations is difficult due to highly screwed operation loads, no support to collect workload statistics, and high complexity of the scheduling problem. So current implementations adopt simple strategies, leading to poor load balance. To address these difficulties, we design an algorithm to schedule operations based on the key distribution of intermediate pairs. The algorithm involves a sub-program for selecting operations for task slots, and we name it the Balanced Subset Sum (BSS) problem. We discuss properties of BSS and design exact and approximation algorithms for it. To transparently incorporate these algorithms into MapReduce, we design a communication mechanism to collect statistics, and a pipeline within Reduce tasks to increase resource utilization. To the best of our knowledge, this is the first work on scheduling MapReduce workload at this fine-grained level. Experiments on PUMA [T+12] benchmarks show consistent performance improvement. The job duration can be reduced by up to 37%, compared with standard MapReduce.

💡 Research Summary

The paper tackles a long‑standing performance bottleneck in the MapReduce programming model: the imbalance of work among individual map or reduce operations. While prior research has largely focused on coarse‑grained scheduling—adjusting the number of map or reduce tasks, replicating data, or re‑partitioning large blocks—this approach ignores the fact that each invocation of the user‑defined map or reduce function (hereafter called an “operation”) can have dramatically different workloads depending on the distribution of keys it processes. When a few operations dominate the amount of intermediate key‑value pairs, the overall job duration is dictated by the slowest task slot, leading to under‑utilized resources and longer runtimes.

To address this, the authors propose a fine‑grained scheduling framework that bases decisions on the key distribution of intermediate pairs. The core idea is to collect, before scheduling, a statistical profile of how many keys each operation will handle. This profile is then used to solve a combinatorial allocation problem they name the Balanced Subset Sum (BSS) problem. Formally, given a set of operations O = {o₁,…,oₙ} and a weight wᵢ for each operation (the estimated number of keys), the goal is to partition O into k subsets (corresponding to the k task slots on a node or across the cluster) such that the sum of weights in each subset is as close as possible to the ideal average load W/k. The BSS problem is shown to be NP‑hard, extending the classic Subset Sum problem to a multi‑target setting.

The paper contributes two families of algorithms for BSS. The first is an exact dynamic‑programming solution that enumerates all achievable weight sums up to a bounded limit. Although exponential in the worst case, it is practical when the number of operations per node is modest or when the weight range is limited (e.g., after scaling). The second family consists of polynomial‑time approximation algorithms. By quantizing weights into buckets of size ε·W/k and applying a greedy “largest‑first” assignment, the algorithm guarantees a solution within (1+ε) of the optimal load balance. The authors provide theoretical bounds on the approximation ratio and empirically demonstrate that even with ε = 0.1 the resulting load imbalance is negligible.

Collecting the key‑distribution statistics is non‑trivial because standard Hadoop does not expose per‑operation metrics. The authors extend the Map phase with a lightweight local counter that records the frequency of each emitted key. At the end of the map task, these counters are serialized into a special metadata channel and sent to a central scheduler before the reduce phase begins. The communication overhead is bounded by the number of distinct keys per mapper, which is typically far smaller than the total number of records. To avoid idle CPU cycles while waiting for the scheduler’s decisions, the Reduce tasks are organized into a pipeline: as soon as a reduce operation finishes processing its assigned keys, the next operation (selected by the BSS algorithm) is launched on the same slot. This pipeline maximizes CPU and memory utilization without requiring additional threads.

The experimental evaluation uses the PUMA benchmark suite, which includes a variety of workloads (e.g., word count, graph processing, join operations) with deliberately skewed key distributions. The authors compare three configurations: (1) vanilla Hadoop’s default round‑robin task assignment, (2) Hadoop with the exact BSS scheduler, and (3) Hadoop with the ε‑approximation BSS scheduler. Results show an average job‑completion time reduction of 22 % relative to the baseline, and up to 37 % for the most skewed workloads. The variance of per‑slot execution times drops from 45 % (baseline) to under 12 % with the BSS scheduler, confirming a much tighter load balance. Overhead analysis reveals that statistics collection and scheduler communication consume less than 5 % of total runtime, validating the practicality of the approach.

In summary, the paper makes four key contributions: (i) it defines the fine‑grained operation‑level scheduling problem for MapReduce and formalizes it as the Balanced Subset Sum problem; (ii) it provides both exact and provably near‑optimal approximation algorithms for BSS; (iii) it designs a low‑overhead mechanism to gather per‑operation key distribution statistics and integrates the scheduler transparently into the existing MapReduce execution pipeline; and (iv) it demonstrates, via extensive experiments on realistic benchmarks, that the approach yields substantial reductions in job duration and improves resource utilization. The work opens several avenues for future research, including adaptive re‑scheduling for dynamic workloads, distributed implementations of the BSS solver for very large clusters, and extensions to other data‑flow systems such as Apache Spark or Flink.