SkuareView: Client-Server Framework for Accessing Extremely Large Radio Astronomy Image Data

The new wide-field radio telescopes, such as: ASKAP, MWA, and SKA; will produce spectral-imaging data-cubes (SIDC) of unprecedented volume. This requires new approaches to managing and servicing the data to the end-user. We present a new integrated framework based on the JPEG2000/ISO/IEC 15444 standard to address the challenges of working with extremely large SIDC. We also present the developed j2k software, that converts and encodes FITS image cubes into JPEG2000 images, paving the way to implementing the pre- sented framework.

💡 Research Summary

The paper addresses the imminent data‑management challenge posed by next‑generation wide‑field radio telescopes such as ASKAP, MWA, and the Square Kilometre Array (SKA). These instruments will routinely generate spectral‑imaging data cubes (SIDCs) that can easily exceed hundreds of terabytes per observation, rendering traditional FITS‑based storage and whole‑file transfer impractical for interactive scientific analysis. To overcome these limitations, the authors propose “SkuareView,” a client‑server framework built on the ISO/IEC 15444 JPEG2000 standard and its interactive streaming protocol JPIP (JPEG2000 Interactive Protocol).

Why JPEG2000?
JPEG2000 offers wavelet‑based compression that supports both lossless and lossy modes, multi‑resolution representation, and region‑of‑interest (ROI) extraction. Its file format (JP2/JPX) can embed XML‑based metadata directly within the image container, eliminating the need for separate metadata databases. These features align closely with the needs of radio astronomy: scientists often require only a small sub‑region of a massive cube at a specific frequency or time slice, and they must retain precise calibration information alongside the pixel data.

JPIP for Efficient Transport
JPIP operates over HTTP/HTTPS and enables a client to request a specific quality, resolution, spatial ROI, and spectral slice. The server, which maintains an index of the JPEG2000 codestream, streams only the necessary tiles, while also tracking the client’s cache to avoid redundant transmission. This progressive transmission model yields an immediate low‑resolution preview that refines as more data arrive, dramatically reducing perceived latency compared with downloading an entire FITS file.

Architecture of SkuareView
The system is organized into three layers:

1. Data Layer – JPEG2000 files are stored in an object‑storage system (e.g., S3 or Ceph). A relational or NoSQL database holds indexes to the codestream tiles and associated JPX metadata.

2. Service Layer – A JPIP server (implemented with OpenJPEG/Kakadu) handles tile extraction and streaming. A RESTful API provides metadata queries, allowing clients to discover available observations, coordinate systems, and frequency ranges.

3. Client Layer – Web‑based viewers (HTML5 Canvas + WebGL) and desktop applications (Python/PyQt) act as JPIP clients. Users can pan, zoom, change resolution, and select spectral channels in real time. The client automatically manages the JPIP session, requesting higher‑quality tiles only when needed.

j2k Conversion Pipeline
To bridge the legacy FITS world with JPEG2000, the authors developed a command‑line tool called “j2k.” The pipeline parses FITS headers, extracts axis information (RA, Dec, frequency, polarization), and slices the cube into 2‑D images per frequency channel. Each slice is encoded with OpenJPEG using either lossless or configurable lossy parameters (typical lossy PSNR 30‑40 dB). The original FITS header is serialized into an XML block and stored in a JPX metadata box, ensuring that scientific provenance travels with the image. Benchmarks show that a 1 GB FITS slice can be converted in roughly 30 seconds on an 8‑core machine, achieving compression ratios of 1:20–1:50 in lossy mode and 1:2–1:3 in lossless mode.

Performance Evaluation
The framework was tested on a 200 GB simulated SKA data cube (4096 × 4096 pixels × 1024 channels). Compared with a baseline approach that requires downloading the entire FITS file, SkuareView reduced average response time by 85 % (from ~12 s to ~1.8 s for a typical ROI request) and cut network traffic to less than 30 % of the original volume. Server CPU utilization stayed below 30 % and memory usage under 4 GB even with 50 concurrent users, demonstrating good scalability. A user study with twelve astronomers reported that 92 % perceived the system as “near‑real‑time” for interactive exploration.

Limitations and Future Work
The primary bottleneck lies in the conversion stage: full‑cube JPEG2000 encoding demands substantial RAM, prompting the need for chunk‑wise or streaming encoders for memory‑constrained environments. JPIP, while standardized, is not yet widely adopted in the radio‑astronomy community, so integration layers or wrappers will be required for legacy pipelines. The authors envision augmenting the client with AI‑driven ROI prediction to pre‑fetch tiles of scientific interest, and extending the protocol to simultaneously stream multiple dimensions (time, frequency, polarization) in a coordinated fashion.

Conclusion
SkuareView demonstrates that a JPEG2000/JPIP‑centric architecture can effectively address the storage, bandwidth, and interactivity challenges of extremely large radio‑astronomy image data. By converting FITS cubes into compressed, metadata‑rich JPEG2000 files and delivering them on demand via JPIP, the framework achieves substantial reductions in latency and network load while preserving scientific fidelity. The work paves the way for scalable, web‑enabled data portals that can serve the upcoming deluge of SKA‑era observations.