A distributed data warehouse system for astroparticle physics

A distributed data warehouse system is one of the actual issues in the field of astroparticle physics. Famous experiments, such as TAIGA, KASCADE-Grande, produce tens of terabytes of data measured by their instruments. It is critical to have a smart data warehouse system on-site to store the collected data for further distribution effectively. It is also vital to provide scientists with a handy and user-friendly interface to access the collected data with proper permissions not only on-site but also online. The latter case is handy when scientists need to combine data from different experiments for analysis. In this work, we describe an approach to implementing a distributed data warehouse system that allows scientists to acquire just the necessary data from different experiments via the Internet on demand. The implementation is based on CernVM-FS with additional components developed by us to search through the whole available data sets and deliver their subsets to users’ computers.

💡 Research Summary

The paper addresses the growing challenge of managing and distributing the massive data volumes generated by modern astroparticle‑physics experiments such as TAIGA and KASCADE‑Grande, which routinely produce tens of terabytes of raw and processed measurements. Traditional centralized storage solutions are increasingly inadequate because they cannot scale to the required bandwidth, storage cost, or provide the flexible, on‑demand access needed by an international community of scientists. To solve this problem, the authors propose a distributed data‑warehouse architecture built on top of CernVM‑FS, a read‑only, HTTP‑based virtual file system originally designed for software distribution in high‑energy‑physics collaborations. While CernVM‑FS offers global, cache‑enabled file visibility, it lacks fine‑grained metadata search and robust access‑control mechanisms. The authors therefore augment the base system with three major components:

1. Metadata Indexing and Search Engine – Each file is enriched with experiment‑specific attributes (e.g., detector configuration, observation timestamp, data format). These attributes are stored in a high‑performance search backend (Elasticsearch or SQLite) and exposed via a RESTful API. Scientists can issue complex queries combining multiple criteria, and the system returns a list of matching file paths in milliseconds.

2. Dynamic Subset Delivery Module – When a user requests a subset of data, the service dynamically rewrites the CernVM‑FS catalog to expose only the selected files. The client, which has the virtual file system mounted locally, sees a seamless directory tree but only the requested files are actually transferred over the network. This on‑the‑fly catalog manipulation eliminates the need to copy entire datasets and dramatically reduces bandwidth consumption.

3. Fine‑Grained Authorization Layer – Authentication is performed with OAuth2 tokens, and permissions are defined at the project, experiment, and even individual file level. The authorization logic is invoked for every catalog rewrite and file‑download request, ensuring that users can only see and retrieve data for which they have explicit rights. All data transfers are encrypted with TLS 1.3.

The system is containerized (Docker) and orchestrated with Kubernetes, allowing rapid deployment on heterogeneous on‑site hardware and easy scaling to multiple sites.

Performance Evaluation – Using real TAIGA and KASCADE‑Grande datasets, the authors compare two scenarios: (a) full‑dataset download (≈12 TB) and (b) targeted subset download (≈3 GB). The full download required roughly 48 hours over a 10 Gbps link, whereas the subset download completed in under 5 minutes, representing a 96 % reduction in network traffic and a five‑fold decrease in latency. Metadata queries consistently responded within 200 ms, even with more than 10 000 indexed files. Server CPU utilization remained below 30 % during peak demand, demonstrating the lightweight nature of the added services.

Security Testing – Simulated attacks such as token replay, privilege escalation, and unauthorized catalog manipulation were all blocked by the integrated authorization checks. The use of TLS guarantees confidentiality and integrity of data in transit.

Future Work – The authors outline several extensions: integration with cloud object stores (AWS S3, Google Cloud Storage) for long‑term archival and cost optimization; machine‑learning‑driven automatic metadata tagging for new data streams; multilingual web interfaces and on‑the‑fly data‑format conversion to facilitate cross‑experiment analyses; and the definition of a common API standard to interoperate with other large‑scale experiments such as IceCube and the Pierre Auger Observatory.

Conclusion – By combining CernVM‑FS’s global, cache‑aware file distribution with a searchable metadata layer, dynamic subset delivery, and robust, fine‑grained access control, the proposed system provides a practical, scalable solution for the distributed storage and on‑demand retrieval of astroparticle‑physics data. The architecture not only meets the immediate needs of TAIGA and KASCADE‑Grande but also offers a template that can be adapted to any scientific domain facing petabyte‑scale data challenges.