Information Systems Playground - The Target Infrastructure, Scaling Astro-WISE into the Petabyte range

The Target infrastructure has been specially built as a storage and compute infrastructure for the information systems derived from Astro-WISE. This infrastructure will be used by several applications that collaborate in the area of information systems within the Target project. It currently consists of 10 PB of storage and thousands of computational cores. The infrastructure has been constructed based on the requirements of the applications. The storage is controlled by the Global Parallel File System of IBM. This file system takes care of the required flexibility by combining storage hardware with different characteristics into a single file system. It is also very scalable, which allows the system to be extended into the future, while replacing old hardware with new technology.

💡 Research Summary

The paper describes the design, implementation, and early operation of the Target infrastructure, a large‑scale storage and compute platform built to support Astro‑WISE and a suite of related scientific and artificial‑intelligence applications. At the time of writing the system provides roughly 10 PB of storage and thousands of CPU cores, and it is architected to scale beyond the petabyte range while allowing hardware refreshes with minimal service disruption.

A set of representative use cases is presented: the KIDS and VIKING optical/infrared surveys processed by Astro‑WISE, the LOFAR Long‑Term Archive (LTA) which will ingest more than a petabyte per year of radio‑interferometer data, the Monk project for handwritten‑text recognition, and the LifeLines biobank. These applications differ markedly in data size (from megabyte‑scale files to terabyte‑scale images), access patterns (daily, monthly, or yearly), and metadata requirements, but they all share a clear separation between bulk data files and relational metadata.

To meet these heterogeneous requirements the Target team selected IBM’s Global Parallel File System (GPFS) as the core file‑system layer. GPFS can aggregate heterogeneous storage pools—high‑performance SSDs, large‑capacity HDDs, and low‑cost tape—into a single POSIX‑compatible namespace. It offers automatic data lifecycle management, mirroring for high availability, and parallel I/O optimized for both streaming large files and handling millions of small file operations. The metadata tier runs on an Oracle 11g RAC cluster with GoldenGate replication to ensure consistency across nodes.

The paper also compares GPFS with Lustre, highlighting GPFS’s advantages in vendor support, hardware‑software independence, and built‑in lifecycle tools, which are critical for a long‑lived, publicly funded infrastructure. Operational integration includes linking the storage cluster to the university’s 3 220‑core compute cluster and an external Grid, as well as embedding the existing Astro‑WISE and LOFAR authentication/authorization mechanisms.

Overall, the Target infrastructure demonstrates that a unified, GPFS‑based platform can simultaneously satisfy the demanding I/O, scalability, reliability, and security needs of diverse scientific workloads, providing a practical blueprint for future petabyte‑scale data archives.