The MAGIC Data Center

The MAGIC I telescope produces currently around 100TByte of raw data per year that is calibrated and reduced on-site at the Observatorio del Roque de los Muchachos (La Palma). Since February 2007 most of the data have been stored and further processed in the Port d’Informacio Cientifica (PIC), Barcelona. This facility, which supports the GRID Tier 1 center for LHC in Spain, provides resources to give the entire MAGIC Collaboration access to the reduced telescope data. It is expected that the data volume will increase by a factor 3 after the start-up of the second telescope, MAGIC II. The project to improve the MAGIC Data Center to meet these requirements is presented. In addition, we discuss the production of high level data products that will allow a more flexible analysis and will contribute to the international network of astronomical data (European Virtual Observatory). For this purpose, we will have to develop a new software able to adapt the analysis process to different data taking conditions, such as different trigger configurations or mono/stereo telescope observations.

💡 Research Summary

**
The paper presents a comprehensive overview of the data handling architecture for the MAGIC I atmospheric Cherenkov telescope and outlines the planned upgrades required to accommodate the forthcoming MAGIC II instrument. Currently, MAGIC I generates roughly 100 TB of raw data per year. After on‑site calibration and a first level of reduction at the Observatorio del Roque de los Muchachos (La Palma), the data are transferred to the Port d’Informació Científica (PIC) in Barcelona. PIC, which also hosts Spain’s Tier‑1 grid center for the Large Hadron Collider, provides a high‑performance computing (HPC) environment and petabyte‑scale storage that the entire MAGIC collaboration can access via grid middleware.

The existing processing pipeline consists of four main stages: (1) high‑speed transfer of raw files over dedicated fiber links with checksum verification; (2) ingestion of extensive metadata (observation time, pointing direction, atmospheric conditions, trigger configuration) into a relational database; (3) calibration and reduction, including gain correction, pedestal subtraction, and extraction of image parameters such as Hillas moments; and (4) automatic generation of higher‑level products (energy spectra, sky maps, light curves). This workflow already supports a distributed analysis model where multiple users can run parallel jobs on the grid.

With the commissioning of MAGIC II, the data volume is expected to increase by a factor of three, and the system must handle both mono‑telescope and stereo observations. The authors argue that a simple scaling of the current pipeline would lead to unacceptable latency and cost. Consequently, they propose three major technical strategies.

First, a scalable storage architecture that combines object‑storage services for “cold” long‑term archives with SSD‑based caches for frequently accessed recent data. This tiered approach optimizes cost while preserving the I/O performance needed for rapid re‑processing. Second, a dynamic workflow engine based on a directed‑acyclic‑graph (DAG) scheduler. The engine will automatically select the appropriate analysis modules according to the trigger settings, energy range, or observation mode, thereby enabling efficient use of both grid and cloud resources. Third, the adoption of a standardized metadata schema aligned with the International Virtual Observatory Alliance (IVOA) recommendations. By exposing the data through VO‑compliant services (SSAP, TAP, and VOTable formats), the MAGIC data products become discoverable and usable within the broader European Virtual Observatory (EVO) network.

To support the production of high‑level data products suitable for the VO, the team plans to implement FITS headers that fully describe the observation conditions and to provide RESTful APIs for query, cutout, and visualization. This will allow external astronomers to combine MAGIC gamma‑ray observations with data from other wavelengths (radio, X‑ray, optical) without bespoke data‑format conversions.

The upgrade roadmap targets a storage capacity of roughly 1 PB and an annual compute budget of about 10 000 CPU‑core‑hours within five years. Software development will focus on modular, plugin‑based components so that new trigger configurations or analysis algorithms can be integrated with minimal code changes. This modularity is expected to reduce long‑term maintenance effort and to empower individual collaboration members to tailor the pipeline to their specific scientific needs.

In summary, the MAGIC Data Center is transitioning from a successful, single‑telescope operation to a robust, multi‑instrument facility capable of handling dramatically larger data rates and more complex analysis workflows. By leveraging grid resources, adopting scalable storage solutions, and aligning with international virtual‑observatory standards, the project aims to provide the entire MAGIC collaboration—and the wider high‑energy astrophysics community—with fast, reliable access to calibrated data and advanced analysis tools, thereby fostering multi‑wavelength studies and accelerating scientific discovery.