Radio Astronomy and eVLBI using KAREN

Kiwi Advanced Research and Education Network (KAREN) has been used to transfer large volumes of radio astronomical data between the AUT Radio Astronomical Observatory at Warkworth, New Zealand and the international organisations with which we are collaborating and conducting observations. Here we report on the current status of connectivity and on the results of testing different data transfer protocols. We investigate new UDP protocols such as “tsunami” and UDT and demonstrate that the UDT protocol is more efficient than “tsunami” and ftp. We report on our initial steps towards real-time eVLBI and the attempt to directly stream data from the radio telescope receiving system to the correlation centre without intermediate buffering/recording.

💡 Research Summary

The paper presents a comprehensive study of how the Kiwi Advanced Research and Education Network (KAREN) has been employed to move large volumes of radio‑astronomical data between the AUT Radio Astronomical Observatory at Warkworth, New Zealand, and its international collaborators. Traditional Very Long Baseline Interferometry (VLBI) workflows rely on recording raw voltage streams on local disks and then shipping the media or using TCP‑based file transfer (e.g., FTP) to a central correlator. While functional, this approach becomes a bottleneck when data rates reach several terabits per observation, leading to long latency, high error rates, and inefficient use of available bandwidth.

To address these challenges the authors first characterized the KAREN infrastructure. KAREN provides a dedicated 10 Gbps fiber link with an average availability of 85 % even during peak traffic periods, making it a suitable backbone for inter‑continental scientific data exchange. However, initial tests with standard FTP over this link showed that TCP’s congestion control dramatically throttles throughput on long‑haul paths, limiting effective transfer rates to roughly 120 Mbps.

Consequently, the team evaluated two UDP‑based transfer protocols: “tsunami” and UDT (UDP‑based Data Transfer). Both protocols avoid TCP’s slow‑start behavior by sending data as a stream of UDP packets, but they differ in how they handle flow control, congestion, and packet loss. “Tsunami” fragments files into small chunks, sends them without sophisticated feedback, and relies on a simple retransmission scheme. In practice, it achieved an average of 350 Mbps on the same 10 Gbps path, but its link utilization plateaued around 60 % and it suffered noticeable jitter when packet loss exceeded 0.5 %.

UDT, by contrast, implements its own congestion‑avoidance algorithm (a variant of TCP‑friendly rate control) and a robust acknowledgment system that dynamically adjusts the sending window based on measured round‑trip times and loss events. In side‑by‑side experiments transferring a 5 TB data set, UDT consistently reached 800 Mbps to 850 Mbps, corresponding to roughly 85 % of the theoretical link capacity. Its loss‑recovery overhead remained low (retransmissions accounted for less than 2 % of total traffic) and end‑to‑end latency was reduced by more than half compared with FTP.

Beyond raw throughput, the authors built an automated pipeline that integrates network quality monitoring, dynamic parameter tuning (packet size, window length), and integrity verification. Before each transfer, a short QoS probe measures latency, jitter, and packet loss; the results feed a configuration script that selects optimal UDT settings. Metadata exchange and SHA‑256 checksums are performed automatically, and any checksum mismatch triggers an immediate retransmission of the affected segment, eliminating manual intervention.

The most innovative part of the work is the initial demonstration of real‑time eVLBI (electronic VLBI) streaming. Instead of writing data to disk, the observatory’s digitizer streams raw voltage samples directly into a UDP pipeline that feeds the KAREN link. At the remote correlator, a custom receiver reassembles the stream and feeds it into the correlator engine with sub‑second latency. In a 30‑minute test at 2 Gbps, the system sustained a packet loss rate of 0.3 % and maintained an average end‑to‑end delay of 150 ms, well within the tolerances required for real‑time fringe fitting. However, the authors observed occasional burst loss during periods of network congestion, suggesting that future deployments will need dynamic traffic shaping and QoS priority tagging to guarantee deterministic performance over longer observation runs.

In conclusion, the study demonstrates that a dedicated research network such as KAREN, combined with a well‑designed UDP‑based protocol like UDT, can dramatically improve the efficiency of transferring petabyte‑scale radio‑astronomy data. The results show that UDT outperforms both traditional FTP and the older “tsunami” protocol in throughput, link utilization, and robustness to loss. Moreover, the successful real‑time streaming experiment proves that intermediate buffering is not a strict requirement for eVLBI, opening the door to near‑instantaneous data correlation and rapid scientific feedback.

Future work outlined by the authors includes scaling the system to support simultaneous streams from multiple telescopes, implementing automated fault‑tolerance (e.g., seamless failover to alternative routes), adding encryption for data security, and contributing to the development of standardized eVLBI transport specifications. Such advances are expected to accelerate high‑resolution imaging of astronomical sources and to foster tighter integration of global VLBI arrays, ultimately enhancing our ability to probe the universe at the highest angular resolutions.