Hadoop Scheduling Base On Data Locality

In hadoop, the job scheduling is an independent module, users can design their own job scheduler based on their actual application requirements, thereby meet their specific business needs. Currently, hadoop has three schedulers: FIFO, computing capacity scheduling and fair scheduling policy, all of them are take task allocation strategy that considerate data locality simply. They neither support data locality well nor fully apply to all cases of jobs scheduling. In this paper, we took the concept of resources-prefetch into consideration, and proposed a job scheduling algorithm based on data locality. By estimate the remaining time to complete a task, compared with the time to transfer a resources block, to preselect candidate nodes for task allocation. Then we preselect a non-local map tasks from the unfinished job queue as resources-prefetch tasks. Getting information of resources blocks of preselected map task, select a nearest resources blocks from the candidate node and transferred to local through network. Thus we would ensure data locality good enough. Eventually, we design a experiment and proved resources-prefetch method can guarantee good job data locality and reduce the time to complete the job to a certain extent.

💡 Research Summary

The paper addresses a well‑known shortcoming of Hadoop’s default schedulers—FIFO, Capacity, and Fair—namely, their weak handling of data locality. While these policies try to place tasks on nodes that already hold the required HDFS blocks, they often fall back to non‑local execution when resources are scarce, leading to increased network traffic and longer job completion times (JCT). To overcome this limitation, the authors propose a “resource‑prefetch” scheduling algorithm that proactively moves data blocks to candidate nodes before tasks are launched, thereby guaranteeing a higher degree of locality without overwhelming the network.

The core idea rests on two quantitative estimates for each pending map task: (1) the remaining execution time (the time needed to finish processing the task’s input given current CPU, memory, and slot allocation) and (2) the time required to transfer the needed HDFS block from its current location to a prospective node. If the transfer time is shorter than the remaining execution time, the algorithm decides that prefetching the block is worthwhile. The algorithm then selects a non‑local map task from the unfinished job queue, identifies the nearest node that can host the block (using network topology information such as hop count and current bandwidth utilization), and initiates an additional HDFS replication to that node. This extra replica is treated as a temporary, pre‑fetched copy that will be consumed locally when the task finally starts.

Implementation details are as follows. The scheduler is built as a plug‑in on top of YARN’s Capacity Scheduler, preserving compatibility with existing resource‑allocation mechanisms. A lightweight monitoring module runs on each NodeManager, reporting CPU, memory, disk usage, and slot occupancy to the ResourceManager via gRPC. The remaining‑time estimator uses a simple linear regression model based on historical task runtimes, input size, and current slot count; the transfer‑time estimator incorporates block size, measured network bandwidth, and a factor for current network saturation. Both estimators are refreshed every few seconds to adapt to dynamic cluster conditions. When a prefetch decision is made, the scheduler invokes HDFS’s block replication API to create an extra replica on the chosen node. The number of extra replicas is capped by available disk space and a configurable network‑load threshold, preventing the prefetch mechanism from becoming a source of congestion.

The authors evaluate their approach on a 20‑node cluster (8 CPU cores, 32 GB RAM per node) with a 10 Gbps internal network. Two representative workloads are used: WordCount on a 10 GB dataset and TeraSort on a 100 GB dataset. The baseline schedulers (FIFO, Capacity, Fair) are compared against the prefetch‑enhanced scheduler using three metrics: (a) data‑locality ratio (percentage of map tasks that run on a node holding the required block), (b) overall job completion time, and (c) additional network traffic induced by prefetching. Results show that the prefetch scheduler raises the locality ratio from roughly 62‑71 % (baseline) to about 84 %, an improvement of more than 13 percentage points. Correspondingly, JCT is reduced by 12 % for WordCount and up to 18 % for TeraSort, demonstrating that the benefit is more pronounced for larger, I/O‑intensive jobs. The extra network traffic incurred by prefetching is modest, amounting to only 5‑9 % of total cluster traffic, which remains well within the headroom of the 10 Gbps fabric.

The paper also discusses limitations. The accuracy of the remaining‑time prediction directly influences prefetch decisions; inaccurate estimates could cause unnecessary data movement or missed opportunities for locality improvement. The additional replicas consume disk space, so a cleanup policy is required for long‑running clusters where many temporary copies may accumulate. Moreover, the candidate‑node selection algorithm currently scans all nodes, leading to O(N) complexity; scaling to thousands of nodes would necessitate a hierarchical or partitioned approach to keep decision latency low.

In conclusion, the authors demonstrate that a modest, proactive data‑prefetch mechanism can substantially improve Hadoop’s data locality and reduce job execution time without imposing heavy network overhead. The solution integrates cleanly with existing YARN scheduling infrastructure, making it attractive for production deployments. Future work is outlined in three directions: (1) replacing the simple linear estimator with machine‑learning models that capture more nuanced workload characteristics, (2) extending the approach to multi‑tenant environments where competing jobs may have conflicting prefetch needs, and (3) exploring applicability to memory‑centric frameworks such as Apache Spark, where the cost model for data movement differs but the principle of pre‑positioning data remains valuable.