Towards Zero-Overhead Adaptive Indexing in Hadoop

Several research works have focused on supporting index access in MapReduce systems. These works have allowed users to significantly speed up selective MapReduce jobs by orders of magnitude. However, all these proposals require users to create indexes upfront, which might be a difficult task in certain applications (such as in scientific and social applications) where workloads are evolving or hard to predict. To overcome this problem, we propose LIAH (Lazy Indexing and Adaptivity in Hadoop), a parallel, adaptive approach for indexing at minimal costs for MapReduce systems. The main idea of LIAH is to automatically and incrementally adapt to users’ workloads by creating clustered indexes on HDFS data blocks as a byproduct of executing MapReduce jobs. Besides distributing indexing efforts over multiple computing nodes, LIAH also parallelises indexing with both map tasks computation and disk I/O. All this without any additional data copy in main memory and with minimal synchronisation. The beauty of LIAH is that it piggybacks index creation on map tasks, which read relevant data from disk to main memory anyways. Hence, LIAH does not introduce any additional read I/O-costs and exploit free CPU cycles. As a result and in contrast to existing adaptive indexing works, LIAH has a very low (or invisible) indexing overhead, usually for the very first job. Still, LIAH can quickly converge to a complete index, i.e. all HDFS data blocks are indexed. Especially, LIAH can trade early job runtime improvements with fast complete index convergence. We compare LIAH with HAIL, a state-of-the-art indexing technique, as well as with standard Hadoop with respect to indexing overhead and workload performance.

💡 Research Summary

The paper addresses a fundamental limitation of existing indexing techniques for Hadoop‑based MapReduce systems: they require indexes to be built ahead of time. In many scientific, social‑media, and other exploratory workloads the query patterns evolve unpredictably, making upfront index design costly or even infeasible. To overcome this, the authors introduce LIAH (Lazy Indexing and Adaptivity in Hadoop), a novel framework that creates clustered indexes incrementally and automatically as a by‑product of normal MapReduce execution.

Core idea – When a map task reads an HDFS block into memory, LIAH re‑uses the same memory buffer to sort the block and build a clustered index on the key(s) used by the job. Because the data is already being read from disk, no extra I/O is incurred. Index construction is performed on otherwise idle CPU cycles and is parallelised across all map tasks in the cluster. Consequently, the first job that triggers indexing experiences virtually no overhead (2‑5 % in the authors’ experiments), while subsequent jobs benefit from the newly created indexes.

Two‑phase indexing policy –

1. Lazy phase – For each block processed by a map task, LIAH checks whether an index already exists. If not, it creates one after the map task finishes, without blocking the job.
2. Adaptive phase – As more jobs run, the proportion of indexed blocks grows until the entire dataset is covered. Index creation is distributed across nodes, avoiding hotspot formation.

Metadata and synchronization – Index descriptors are stored as extended attributes in HDFS file metadata. Map tasks consult this metadata only when they start reading a block, eliminating complex coordination, locks, or a central index manager. LIAH therefore integrates seamlessly with the existing Hadoop scheduler.

Experimental evaluation – The authors compare LIAH against HAIL (a state‑of‑the‑art pre‑built indexing approach) and vanilla Hadoop on a 10‑node cluster using several selective query workloads. Results show:

• Initial indexing overhead of LIAH is negligible (≈ 2‑5 % of job runtime).
• Once indexes exist, query execution time improves by up to an order of magnitude.
• LIAH converges to a fully indexed dataset faster than HAIL because it trades a tiny amount of extra CPU work early on for rapid index propagation.

Limitations and future work –

• Index construction is CPU‑intensive; in CPU‑saturated environments the benefit may diminish.
• Current implementation supports only single‑column clustered indexes; extending to multi‑column or secondary indexes will require additional design.
• Query planners must be aware of partially indexed datasets and decide dynamically whether to use an index or fall back to a full scan.

Conclusion – LIAH demonstrates that adaptive, zero‑overhead indexing is feasible in Hadoop. By piggy‑backing index creation on the inevitable data reads of map tasks, it eliminates the need for costly upfront index preparation while still delivering dramatic performance gains for selective workloads. This approach is especially valuable for domains where query patterns are unknown a priori, offering a practical path toward self‑optimising big‑data processing pipelines.