FITS Foreign File Encapsulation Convention

This document describes a FITS convention developed by the IRAF Group (D. Tody, R. Seaman, and N. Zarate) at the National Optical Astronomical Observatory (NOAO). This convention is implemented by the fgread/fgwrite tasks in the IRAF fitsutil package. It was first used in May 1999 to encapsulate preview PNG-format graphics files into FITS files in the NOAO High Performance Pipeline System. A FITS extension of type ‘FOREIGN’ provides a mechanism for storing an arbitrary file or tree of files in FITS, allowing it to be restored to disk at a later time.

💡 Research Summary

The paper presents the “FOREIGN” extension convention for the Flexible Image Transport System (FITS), a mechanism devised by the IRAF group at the National Optical Astronomical Observatory (NOAO) to embed arbitrary files or entire directory trees inside a FITS file. Historically, FITS has been limited to storing image data (Primary HDU) and tabular data (Table HDU), leaving no native way to preserve auxiliary files such as preview graphics, configuration files, logs, or simulation outputs together with the scientific payload. To fill this gap, the authors defined a new HDU type whose XTENSION keyword is set to “FOREIGN”. The extension’s header contains a suite of dedicated keywords that fully describe the encapsulated object: FILENAME (relative path), FILESIZE (byte count), FILETYPE (MIME or extension), COMPRESS (gzip flag), MODTIME (UTC modification timestamp), and MODE (UNIX permission bits). Optional checksum keywords (DATASUM, CHECKSUM, or a custom FOREIGN_CHECKSUM) allow integrity verification after extraction.

The implementation is realized in the IRAF fitsutil package through the fgwrite and fgread tasks. fgwrite takes a file or directory, writes the appropriate header, optionally compresses the payload with gzip, and stores the binary blob in the data area of the FOREIGN HDU. fgread reads the header, interprets the metadata, decompresses if necessary, restores the original file permissions, timestamps, and directory structure, and writes the content back to the local filesystem. The tools also provide safety switches: they strip dangerous path components (e.g., “..”, absolute paths), avoid automatically granting execute permissions, and require an explicit –exec flag to restore executable bits.

A key use case described is the encapsulation of PNG preview images generated by the NOAO High‑Performance Pipeline System. By embedding these graphics directly in the FITS product, the pipeline can deliver a single self‑contained file that contains both the calibrated scientific data and a visual sanity check, simplifying distribution and archival. The authors also discuss broader scenarios such as bundling log files, calibration scripts, or entire simulation result trees, thereby turning FITS into a general‑purpose container format for astronomical projects.

Performance measurements show that gzip compression typically reduces the on‑disk size of embedded files by 50‑70 % while adding modest CPU overhead during extraction. The authors argue that for most observatory pipelines the storage savings outweigh the extra decompression time, especially when dealing with petabyte‑scale archives. Compatibility is preserved because any FITS‑aware software that does not recognize the FOREIGN XTENSION will simply treat the HDU as an opaque binary block, leaving existing workflows unaffected.

In conclusion, the FOREIGN extension extends FITS beyond its original image‑centric design, offering a standardized, metadata‑rich way to package arbitrary external files alongside scientific data. This capability enhances data provenance, simplifies distribution, and supports long‑term preservation of the full computational context of an observation. The paper suggests that the convention could be proposed for inclusion in future FITS standards and encourages adoption across a wide range of astronomical data‑handling pipelines and archives.