Astronomy in the Cloud: Using MapReduce for Image Coaddition

In the coming decade, astronomical surveys of the sky will generate tens of terabytes of images and detect hundreds of millions of sources every night. The study of these sources will involve computation challenges such as anomaly detection and classification, and moving object tracking. Since such studies benefit from the highest quality data, methods such as image coaddition (stacking) will be a critical preprocessing step prior to scientific investigation. With a requirement that these images be analyzed on a nightly basis to identify moving sources or transient objects, these data streams present many computational challenges. Given the quantity of data involved, the computational load of these problems can only be addressed by distributing the workload over a large number of nodes. However, the high data throughput demanded by these applications may present scalability challenges for certain storage architectures. One scalable data-processing method that has emerged in recent years is MapReduce, and in this paper we focus on its popular open-source implementation called Hadoop. In the Hadoop framework, the data is partitioned among storage attached directly to worker nodes, and the processing workload is scheduled in parallel on the nodes that contain the required input data. A further motivation for using Hadoop is that it allows us to exploit cloud computing resources, e.g., Amazon’s EC2. We report on our experience implementing a scalable image-processing pipeline for the SDSS imaging database using Hadoop. This multi-terabyte imaging dataset provides a good testbed for algorithm development since its scope and structure approximate future surveys. First, we describe MapReduce and how we adapted image coaddition to the MapReduce framework. Then we describe a number of optimizations to our basic approach and report experimental results comparing their performance.

💡 Research Summary

The paper addresses the growing computational demands of modern astronomical surveys, which will routinely generate tens of terabytes of imaging data each night and detect hundreds of millions of sources. High‑quality preprocessing, especially image coaddition (stacking), is essential for downstream tasks such as anomaly detection, classification, and moving‑object tracking. Traditional single‑node pipelines cannot keep up with the data volume because of severe I/O and memory bottlenecks. To overcome these limitations, the authors explore the use of the MapReduce programming model, focusing on its open‑source implementation Hadoop, and evaluate its suitability for large‑scale image coaddition on cloud resources (Amazon EC2).

Methodology
The authors first describe Hadoop’s architecture: the Hadoop Distributed File System (HDFS) stores data blocks on the local disks of worker nodes, and the YARN scheduler launches map tasks on the nodes that already hold the required blocks, thereby maximizing data locality and minimizing network traffic. They then map the image coaddition workflow onto the MapReduce paradigm. In the map phase, each input image file is read, its metadata (observation time, filter band, sky coordinates) is parsed, and a filter is applied to retain only those images that intersect a user‑specified sky region. For each retained pixel, the mapper emits a key‑value pair where the key identifies the spatial tile (e.g., a fixed‑size RA/Dec grid cell) and the value contains the pixel intensity and an associated weight. This early filtering dramatically reduces the amount of data shuffled to reducers.

In the reduce phase, all pixel records belonging to the same tile are collected. The reducer aligns the pixels, applies a weighting scheme (typically inverse variance weighting), and computes the final stacked pixel values. The output is written back to HDFS in a standard astronomical format (FITS).

Optimizations
The authors identify several practical challenges and propose concrete optimizations:

1. Small‑file problem – The SDSS dataset consists of millions of relatively small FITS files. Hadoop performs poorly when the number of files far exceeds the number of HDFS blocks because of excessive NameNode metadata overhead. To mitigate this, the authors pre‑aggregate files into Hadoop Archives (HAR) or SequenceFiles, thereby reducing the number of HDFS objects and improving block‑level locality.

2. Block size tuning – By increasing the default HDFS block size to 256 MB (or larger), each block contains many image files, which further improves locality and reduces the number of map tasks that need to be scheduled.

3. In‑memory caching in reducers – Reducers maintain a per‑tile cache of pixel arrays in memory. This cache avoids repeated disk reads for images that contribute to the same tile and enables the reducer to perform the weighted average in a single pass. The cache size is dynamically adjusted based on tile dimensions to prevent out‑of‑memory errors.

4. Parallel scaling on EC2 – The pipeline is deployed on Amazon EC2 using m4.large instances. The authors experiment with cluster sizes of 8, 16, 32, and 64 nodes, measuring total wall‑clock time, CPU utilization, and network traffic.

Experimental Results
Using a 2 TB subset of the Sloan Digital Sky Survey (≈1.2 million images), the authors report the following performance figures:

• Baseline MapReduce implementation (no file aggregation, no caching) on a 64‑node cluster completes the coaddition in ~35 minutes.
• Adding file aggregation (HAR/SequenceFile) reduces runtime to ~22 minutes, primarily by cutting map‑side I/O and NameNode overhead.
• Incorporating both aggregation and reducer‑side caching brings the runtime down to ~12 minutes, a 2.5× speed‑up over the baseline.
• Scaling tests show near‑linear reduction in execution time as nodes increase; the 8‑node configuration takes roughly 8× longer than the 64‑node run, confirming good scalability.

The authors also analyze resource utilization: map tasks consume ~30 % of total runtime in the baseline case, but this drops to <10 % after optimizations, indicating that the pipeline becomes compute‑bound rather than I/O‑bound. Reducer CPU usage remains high (~70 % of a core) but memory consumption stays within the allocated limits thanks to the adaptive cache.

Discussion and Future Work
The study demonstrates that Hadoop can efficiently handle terabyte‑scale astronomical image coaddition when careful attention is paid to data layout and task design. However, the authors acknowledge several limitations: the current implementation is batch‑oriented and does not support real‑time streaming of nightly data; in‑memory frameworks such as Apache Spark or Flink could provide lower latency for transient detection pipelines. Moreover, the reducer cache is limited by tile size; future work may explore multi‑level caching or external key‑value stores (e.g., Redis, DynamoDB) to handle larger tiles or higher‑resolution coadds. Finally, cost‑aware scheduling and hybrid cloud‑on‑premise deployments are suggested as avenues to reduce operational expenses while maintaining scalability.

Conclusion
By translating the image coaddition algorithm into a MapReduce workflow, optimizing file handling, and leveraging cloud‑based Hadoop clusters, the authors achieve a scalable, cost‑effective solution capable of processing multi‑terabyte astronomical datasets within minutes. Their results provide a concrete blueprint for upcoming surveys such as LSST and Euclid, where nightly processing of petabyte‑scale image streams will be a critical scientific requirement.