Automatic Optimized Discovery, Creation and Processing of Astronomical Catalogs

We present the design of a novel way of handling astronomical catalogs in Astro-WISE in order to achieve the scalability required for the data produced by large scale surveys. A high level of automation and abstraction is achieved in order to facilitate interoperation with visualization software for interactive exploration. At the same time flexibility in processing is enhanced and data is shared implicitly between scientists. This is accomplished by using a data model that primarily stores how catalogs are derived; the contents of the catalogs are only created when necessary and stored only when beneficial for performance. Discovery of existing catalogs and creation of new catalogs is done through the same process by directly requesting the final set of sources (astronomical objects) and attributes (physical properties) that is required, for example from within visualization software. New catalogs are automatically created to provide attributes of sources for which no suitable existing catalogs can be found. These catalogs are defined to contain the new attributes on the largest set of sources the calculation of the attributes is applicable to, facilitating reuse for future data requests. Subsequently, only those parts of the catalogs that are required for the requested end product are actually processed, ensuring scalability. The presented mechanisms primarily determine which catalogs are created and what data has to be processed and stored: the actual processing and storage itself is left to existing functionality of the underlying information system.

💡 Research Summary

**
The paper presents a novel architecture for handling astronomical catalogs within the Astro‑WISE information system, specifically designed to meet the scalability demands of modern large‑scale sky surveys that generate billions of sources and thousands of measured attributes. Traditional static relational databases, as used by projects such as SDSS and WFCAM, suffer from rigidity: catalogs are released as fixed releases, re‑processing requires downloading large data volumes, and users must understand internal data representations. Moreover, forward‑chaining approaches store query histories but lack explicit data lineage, limiting reuse and flexibility.

Astro‑WISE addresses these shortcomings by turning catalogs themselves into process targets—objects that encapsulate not only the final data but also the full chain of operations that produce it. Central to this concept is the Source Collection, an abstract representation of a catalog that stores (1) a set of sources identified by unique IDs, (2) a set of attributes (physical quantities), (3) an operator describing the transformation needed to obtain the catalog (e.g., filter, attribute calculation, concatenation, selection), and (4) configurable process parameters. Crucially, a Source Collection can exist without its data being materialized; only its lineage metadata is persisted until a request forces actual computation.

When a scientist issues a data‑pull request—specifying a source base, a selection criterion, and the desired attributes—the system automatically constructs a dependency graph that links the requested final Source Collection back through all required intermediate collections to the raw data. This graph is built backward from the target (target‑processing), allowing the system to determine whether existing catalogs satisfy parts of the request or whether new intermediate collections must be created.

The paper details several key mechanisms:

1. Automatic creation of Source Collections based solely on lineage. New collections are defined even before any data is processed, enabling them to serve as dependencies for downstream targets.

2. Graph optimization through temporary duplication of the dependency graph. The system reorders operations to minimize work—e.g., applying filters before expensive attribute calculations—thereby reducing the volume of data that must be processed.

3. Partial processing of large collections. Only the subset of sources required for the final result is processed; the rest remains untouched. Results that are likely to be reused (e.g., absolute magnitudes computed for a large parent set) are stored persistently, while highly specific intermediate results are kept transient.

4. Logical relationship inference between source sets using an algorithm introduced in a companion paper (Buddelmeijer et al. 2011a). By examining lineage alone, the system can deduce inclusion, intersection, or exclusion relationships, which is essential for avoiding redundant calculations in complex graphs.

5. Decoupling of attribute calculation logic from its application. Scientists can plug in custom calculation modules while the system handles the orchestration, preserving the data‑pull paradigm.

6. Integration with query‑driven visualization tools. Because the data‑pull process is highly automated, a visualization client can request only the data needed for a plot, and Astro‑WISE will deliver it with minimal preprocessing.

The authors illustrate the workflow with a concrete example: a scientist requests apparent and absolute magnitudes for galaxies with redshift < 0.1. Starting from a large Source Collection A (containing apparent magnitudes and redshifts), the system creates a filtered collection B, detects that absolute magnitudes are missing, and defines a generic Attribute Calculator C that can compute them for the entire parent set A. After graph optimization, the filter is applied first, then the calculator runs only on the filtered subset, and the final attributes are concatenated and selected for output. The intermediate results (e.g., absolute magnitudes for the whole A) are stored for future reuse, demonstrating the system’s emphasis on reusability.

From a storage perspective, Astro‑WISE adopts a just‑in‑time materialization policy: data are persisted only when it is beneficial for performance or future reuse; otherwise, they remain virtual. This dramatically reduces storage overhead and I/O traffic, which are critical bottlenecks in petabyte‑scale surveys.

Overall, the paper contributes a functional‑style, data‑lineage‑centric framework that transforms catalog handling from a static, push‑oriented process into a dynamic, request‑driven service. By leveraging object‑oriented programming for metadata, functional concepts for operations, and graph‑based optimization, Astro‑WISE achieves high automation, scalability, and flexibility, positioning it as a powerful backend for next‑generation astronomical data analysis and visualization pipelines.