Archiving multi-epoch data and the discovery of variables in the near infrared

We present a description of the design and usage of a new synoptic pipeline and database model for time series photometry in the VISTA Data Flow System (VDFS). All UKIRT-WFCAM data and most of the VISTA main survey data will be processed and archived by the VDFS. Much of these data are multi-epoch, useful for finding moving and variable objects. Our new database design allows the users to easily find rare objects of these types amongst the huge volume of data being produced by modern survey telescopes. Its effectiveness is demonstrated through examples using Data Release 5 of the UKIDSS Deep Extragalactic Survey (DXS) and the WFCAM standard star data. The synoptic pipeline provides additional quality control and calibration to these data in the process of generating accurate light-curves. We find that 0.6+-0.1% of stars and 2.3+-0.6% of galaxies in the UKIDSS-DXS with K<15 mag are variable with amplitudes \Delta K>0.015 mag

💡 Research Summary

The paper presents a comprehensive design and implementation of a synoptic pipeline and accompanying database schema within the VISTA Data Flow System (VDFS) to handle multi‑epoch near‑infrared survey data. While the original VDFS was built for processing and archiving single‑epoch images from UKIRT‑WFCAM and VISTA main surveys, modern time‑domain astronomy demands the ability to store, cross‑match, and analyse repeated observations of the same sky area. To meet this need, the authors introduce four core tables—SynopticSource, Variability, BestMatch, and auxiliary linking tables—that together allow efficient retrieval of light‑curves, variability statistics, and quality flags for billions of sources.

The synoptic pipeline consists of five stages: (1) image preprocessing (bias, flat‑field, background subtraction, astrometric alignment); (2) precise photometry using PSF fitting and colour/absolute calibration; (3) source matching across epochs via the BestMatch algorithm, which incorporates positional uncertainties and observing‑condition metadata; (4) variability detection employing statistical metrics such as RMS, χ², and the Stetson J index; and (5) construction of final light‑curves with optional merging of external datasets. Quality control is embedded throughout; each detection receives a set of flags (e.g., poor PSF, high background, bad seeing) that are automatically excluded from variability calculations, ensuring robust light‑curve generation.

The system is validated using Data Release 5 of the UKIDSS Deep Extragalactic Survey (DXS) and WFCAM standard‑star observations. In the DXS K‑band (5σ limit ≈ K 20 mag), the pipeline produces reliable light‑curves down to K ≈ 15 mag. Applying a variability threshold of ΔK > 0.015 mag, the authors find that 0.6 ± 0.1 % of stars and 2.3 ± 0.6 % of galaxies exhibit significant variability. The higher variability fraction among galaxies suggests that near‑infrared variability, possibly driven by active nuclei or dust‑obscured processes, is more common than previously recognised in this wavelength regime.

A notable feature of the implementation is its dynamic updating capability. When new epochs are ingested, the BestMatch table is recomputed, and all affected light‑curves are automatically refreshed, providing users with up‑to‑date time series without manual re‑processing. The database runs on a MySQL backend, with pipeline scripts written in Python and Java, and integrates seamlessly with the existing VDFS infrastructure. Its modular schema permits future extensions—such as adding colour‑variability indices, proper‑motion measurements, or spectroscopic cross‑matches—making it adaptable to upcoming surveys like VISTA Variables in the Via Lactea (VVV) and even optical time‑domain projects such as LSST.

In summary, the authors deliver a scalable, end‑to‑end solution for archiving and analysing multi‑epoch near‑infrared data. By coupling a rigorously calibrated photometric pipeline with a purpose‑built relational database, they enable efficient discovery of rare variable and moving objects within the massive data streams produced by modern survey telescopes. The demonstrated performance on UKIDSS‑DXS data confirms the system’s scientific utility and sets a solid foundation for future time‑domain infrared astronomy.