An Interactive Metrics Dashboard for the Keck Observatory Archive

Since 2004, the Keck Observatory Archive (KOA) has operated as a NASA-funded collaboration between the NASA Exoplanet Science Institute ( NExScI) and the W.M. Keck Observatory. It ingests and serves all data acquired by the twin 10-meter Keck telescopes on Mauna Kea, Hawaii. In the past three years, KOA has begun a modernization program to replace the architecture and systems used since the archive’s creation with a new modern Python-based infrastructure. This infrastructure will position KOA to respond to the rapid growth of new and complex data sets that will be acquired by new instruments now in development, and enable follow-up to identify the deluge of alerts of transient sources expected by new survey telescopes such as the Vera C. Rubin Observatory. Since 2022, KOA has ingested new data in near-real time, generally within one minute of creation, and has made them immediately accessible to observers through a dedicated web interface. The archive is now deploying a new, scalable, Python-based, VO-compliant query infrastructure built with the Plotly-Dash framework and R-tree indices to speed-up queries by a factor of 20. The project described here exploits the new query infrastructure to develop a dashboard that will return live metrics on the performance and growth of the archive. These metrics assess the current health of the archive and guide planning future hardware and software upgrades. This single dashboard will enable, for example, monitoring of real-time ingestion, as well as studying the long-term growth of the archive. Current methods of gathering metrics that have been in place since the archive opened will not support the archive as it continues to scale. These methods suffer from high latency, are not optimized for on-demand metrics, are scattered among various tools, and are cumbersome to use.

💡 Research Summary

The Keck Observatory Archive (KOA), a NASA‑funded collaboration between NExScI and the W.M. Keck Observatory, has been ingesting and serving all data from the twin 10‑meter telescopes since 2004. In response to the rapid growth of new instruments and the upcoming flood of transient alerts from facilities such as the Vera C. Rubin Observatory, KOA launched a three‑year modernization effort to replace its legacy C‑based stack with a fully Python‑centric infrastructure. Since 2022 the archive has been capable of ingesting data within roughly one minute of creation and exposing it instantly through a dedicated web portal.

The new infrastructure is built on a scalable, VO‑compliant query service (nexsciTAP) that uses Plotly‑Dash for the front end, Jinja2 for dynamic query templating, and SQLAlchemy for database interaction. R‑tree indices and parallel query strategies accelerate look‑ups by a factor of about 20 compared with the previous system.

The paper describes the development of an interactive metrics dashboard that leverages this query layer to provide live performance and growth statistics. The first release updates every 5–7 seconds, displaying total Level‑0 file counts, instrument‑specific totals, and histograms that can be downloaded as CSV. A loading spinner indicates when the dashboard is refreshing, and pull‑down menus let users filter by instrument, data type, date range, and cadence (weekly or monthly). KPI cards are color‑coded for quick health assessment.

Database design is a key part of the solution. Year‑specific tables store time and instrument columns as primary keys, while science and calibration file counts are defined as virtual columns to keep payloads small. Pre‑computed static summary tables covering 1994‑2024 are generated on daily, monthly, and annual bases, while the current year (2025 at the time of writing) remains dynamic. Parallel queries against the static and dynamic tables dramatically reduce response latency, enabling the seven‑second update cadence.

A backend daemon continuously polls the nexsciTAP service for new metrics, logs each ingestion, and feeds the results to the dashboard. Plotly’s built‑in event monitoring visualizes update speeds in milliseconds, providing operational staff with immediate feedback on system performance.

The authors conclude that the dashboard successfully meets KOA’s need for near‑real‑time monitoring of file ingestion and plan to expand it to include additional metrics such as data volume, query counts, and download traffic. Future enhancements will add overlay plots for selected instruments and other advanced visualizations. By adopting widely used open‑source tools and a modular, parallelized architecture, KOA has built a scalable solution that can keep pace with the anticipated data deluge from next‑generation astronomical facilities.