Astro-WISE Information System

arXiv:1111.6465v1 [astro-ph.IM] 28 Nov 2011 **Volume Title** ASP Conference Series, Vol. **Volume Number** **Author** c⃝**Copyright Year** Astronomical Society of the Pacific Astro-WISE Information System Willem-Jan Vriend, Edwin A. Valentijn, Andrey Belikov, Gijs A. Verdoes Kleijn Kapteyn Astronomical Institute, P.O. Box 800, 9700 AV Groningen, the Netherlands Abstract. Astro-WISE is a scientific information system for the data processing of optical images. In this paper we review main features of Astro-WISE and describe the current status of the system. 1. Introduction Astro-WISE 1 (Valentijn et al. 2007) stands for Astronomical Wide-field Imaging Sys- tem for Europe. The system was initially developed to support the data processing of the Kilo Degree Survey (KiDS) (Verdoes Kleijn et al. 2012) on the VLT (Very Large Telescope) Survey Telescope (VST2). The VST is a 2.61 m diameter imaging telescope installed on the Paranal Observatory in Chile. The instrument installed on the telescope is OmegaCAM, a large format (16k x 16k) CCD camera which will produce up to 15 TB of raw images per year. This amount is multiplied at least by a factor of 3 by the data processing. Astro-WISE was planned as a storage and data processing system for KiDS, but, with time, grew up to a general astronomical information system. It has developed into a much wider data processing information system which can be used in many other disciplines. The idea behind Astro-WISE is to have the data model, data and data processing in a single system. At the same time, such a system should be shared by a number of institutes and sites, as the scientific community working with the data in the system is widely dis- tributed and each group is coming to the system with their own resources (data storage and computing facilities). Moreover, each group is developing a part of the software implemented in the system. The users must be able to form communities (groups) to process the data for the same instrument or project. 2. Basic Principles of the System The development of the Astro-WISE information system started from the very practical challenge: enable a community of researchers distributed over the world to process the 1http://www.astro-wise.org 2http://www.eso.org/public/teles-instr/surveytelescopes/vst.html 1 2 Willem-Jan Vriend, Edwin A. Valentijn, Andrey Belikov, Gijs A. Verdoes Kleijn data of the OmegaCAM 256 Megapixel camera imaging survey. These scientists should be able to evaluate the quality of the data, apply a number of calibrations, share the data in the team and employ distributed resources of PB scale of data storage and Tflops capacity in data processing. From this our basic requirements on the system are derived: * Scalability of the system: any part of the system, i.e. data storage, data pro- cessing, metadata management, should be scalable with the increase of incoming data and a number of users involved in the data processing. The system should be scalable to the data processing algorithms and pipelines allowing to implement new pipelines and reprocess the same data with new algorithms. The scalability to the data mining should be implemented, i.e., the system should provide all possible ranges of requests from the retrieval of a single data item by identifier to a complicated archive study involving multiple complex queries. * Distributed system: the system allows any activity to be distributed among dif- ferent users and different sites where the system is implemented. * Traceability: all activity in the system should leave a clear footprint so that it will be possible to trace the origin of any changes in the data and find an algorithm, program and user who created a data item. * Adaptability. The system allows for a number of different scientific use-cases and provides resources, pipelines and expertise to perform data processing according to the user’s interests. Requirements were set on the common data model realized in the system. The common data model is the core of Astro-WISE and implements the following features: 1. Inheritance of data objects. Using object oriented programming, all objects within the system can inherit key properties of the parent object, all these prop- erties are made persistent (i.e., stored in a database). 2. Full lineage. The linking (associations or references, or joins) between object instances in the database is maintained completely. Each data item in the system can be traced back to its origin. The tracing of the data object can be both forward and backward, for example, this makes it possible to find which raw frames were used to determine magnitudes, shapes and position for this particular source and, at the same time, which sources were extracted on the particular raw frame. 3. Consistency. At each processing step, all processing parameters and the inputs which are used, are kept within the system. Astro-WISE can keep the previous versions of all data items along with all parameters used to prod

This content is AI-processed based on open access ArXiv data.