Radio Astronomy Data Transfer and eVLBI using KAREN

Radio Astronomy Data Transfer and eVLBI using KAREN Stuart Weston 1 , Timothy Natusch 1 and Sergei Gulyaev 1 1 Institute for Radio Astronom y and Space Resear ch, Auc kland University of Technology , Private Bag 92006, Auckland 114 2, New Zealand stuart.weston@aut.ac.nz , tim.natusch@aut.ac.nz , sergei.guly aev@aut.a c.nz Abstract Kiwi Adva nced Research and Educati on Network ( KAREN) has been used to t ransfer la rge volum es of radi o astronomical data between th e Radio Astr onomical Observatory at Wark worth, New Ze aland and various inte rnational organizati ons involve d in joint projects and VLB I observati ons . He re we report on the current status of connectivity and on the result s of test ing differ ent data tra nsfer pr otocols. We investi gate new UDP proto cols such a s ’tsunam i’ and UDT and demonstra te that the UDT pr otocol is m ore efficient than ’ts unami ’ and ‘ftp’. We also re port on the t ests on direct data streami ng from the radi o telescope recei ving system t o the correlation centre without intermediate buffering or recording (real-time eVLBI). 1. Introduction The Warkw orth Radi o Astron omical Obse rvatory (WRAO) i s located s ome 60 km north of the cit y of Auc kland, near the township of Warkworth. The observatory is operated by the Institute for Radio Astronomy and Space Research (IRASR) of the Auckland University of Technology (AUT). The observatory’s 12-m radio telescope operates in three frequency ba nds centered a round 1 .4, 2.4 and 8. 4 GHz. This fast- slewing (5° per sec ond in Azim uth) antenna i s well suited to the purpos es of geodetic VLBI (Very Long Baselin e Interferometry), for spacecraf t na vigatio n and trac king. It is also effectively used for astrono mical VLBI research in conjunction with large radio telescopes [1 ]. Each VLBI sessi on may resul t in many Terabytes of data recorded, so transfer of data to a data processing (correlation) centre is an issue of gr eat imp ortance for VLBI. Previously data was recorded to magnetic tapes, these were then sent via traditional m eans such as post/courier to the c orrelation centre. More r ecently with the reduced c ost and increased capacity of hard dis k storage, the data has be en recorded to rem ovable disk arrays which can then also be sent physicall y to the correl ation centre . Recent ly with the advent of e VLBI the data is sent via high speed networks t o the correlation centers in real-tim e. The IR ASR coll aborates with a num ber of inte rnational part ners. This res earch collabo ration can be br oken into three major res earch topics/groups: • Astrophysical VLBI and eVL BI observa tions in the fram ework of the Australian Long Baseline Array (LBA) for the study of physics and or igin of extragalactic and galac tic radio sources: active g alactic nuclei, radio galaxies, supernova remnant s and star form ation regions . These observatio ns require dat a to be sent from the 1 2-m radio telescope to C SIRO (Sy dney) and/ or to the Curtin Unive rsity , Perth for cor relation and i maging [ 2]. • Observation and navigation of inter-pl anetary spacecraft, as well as ground tracking services for a variety of space missions. In 2010 the 12-m radio telescope partic ipated in VLBI observations of JAXA’s IKAROS and Akatsuki spacecraft; ESA’s Mars E xpress was successfully detected. Data was tran sferred electroni cally (e- transfer) directly to Metsähovi, Finland a nd to the Joint Institute for VLBI in Europe (JIVE) [3]. • Geodetic VLB I and regula r IVS (I nternational V LBI Service f or Geodesy an d Astrom etry) observat ions of a large group of quasars, which uniquely support th e International Celestial Re ference Frame [4]. The observational data are to be sent t o data correlation ce ntres located in the United States Naval Observator y (USNO) and the Max Planck Institute for Ra dio Astronomy (MPIfR) in Bonn (Germany). W ith the connection of the WRAO to KAREN network, our intention is to o ptimize the us e of KAREN for transferring la rge volumes of obs ervational data to our part ners in Austra lia, Asia, Nort h America and Eur ope and for conducting real -time eVLBI. In this paper we discuss t he use of FTP over TCP/IP pr otocol for t ransferring dat a, and compare the perf ormance of n ew protocols w hich are being used in radio a stronom y such as ‘tsunam i’ and UDT (UDP- based Data Tra nsfer) via t he network prot ocol UDP. 2. Network Protocols a nd Connectivity Status Point to point with no hops FTP is efficient, but as th e number of hops in the route increases and the i ncidence of lost packets and collisions increases the TCP congestion avo idance algorithm becomes a severe limitation to the throughput that can be achie ved. ‘Tsun ami’ is an UDP file tran sfer protocol dev eloped by Jan Wagner of the Metshovi Radio Ob servatory in 2007 [5]. This is the protocol of choice for sending files to Bonn for the IVS observations, as stipu lated by the MPIfR. Another UDP pr otocol called UDT was developed in the University of Illinois in 200 5 [6, 7]. UDT was investi gated by our Australian partners (private communication Chris Phillips, CSIRO) in 2008-09. This n ow appears to have matured and is of further interest and warrants further investigation. Table 1 presents the destinati ons that con nectivity has been achieved wi th (colum n 1), the p rotocols that have been verified for data transfer (col um ns 2 a nd 3) and c ommand line access to remo te servers for initiating data transfe rs (colum n 4). Table 1: Connecti vity established bet ween the IRASR and i ts VLBI partners via KAR EN Destination Protocol Command Date CSIRO (Australia) UDP Tsunami, UDT ssh 01/04/2010 Bonn (Ger many) UDP Tsunami, UDT ssh 01/06/2010 JIVE (Nethe rlands) UDP - iperf 27/07/2010 Metsähovi (Finland) UDP Tsunami, UDT ssh 21 /07/2010 USNO (United States) UDP - ssh, iperf 15/01/2011 GSI (Japan ) UDP Tsunami, UDT ssh, iperf 10 /1/02011 3. Data Transfer Tests and Results Tab le 2 presents the results of data transfer tests between the IRASR and data pro cessing centers in Bonn and Metsähovi. Th e results were obtaine d by transfe rring an actua l 16 bit VLBI file produced in observations with the 12-m radio telescope . Colum n 2 shows the p rotocol use d, colum n 3 gives the a mount of data sent in by tes, colum n 4 provides the time it took to transfer the data and co lumn 5 shows an average throughput rate over t he period. T he data was transferred from the IRASR's IBM Blade server via the KAR EN network using the default settin gs for each protocol with no tu nning. The light pat h between New Zealand and Europe is shown i n Figure 1 . Table 2: Data Transfer results: IRASR to Bonn and Metsähovi Route Prot ocol Bytes Time (s) Throughput (Mbps) AUT – Bonn Ftp 65G 8016 65 AUT – Met sähovi Ftp 3.1G 432 61 AUT – Bonn Tsunami 65G 3466 151 AUT – Metsähovi Tsunami 65G 4979 105 AUT – Bonn UDT 65G 1920 273 AUT – Metsähovi UDT 65G 1157 453 Figure 1. The lightpath New Zealand – Metsähovi – JIVE. Satellite image: Blue Marble Ne xt Genera tion. Courtesy: NASA Visi ble Earth The m ain purpose of these tests was t o compare t he performa nce of differe nt data transfe r protoc ols. Data i n Table 2 clearly demonstrate the advantage of the UDT pr otocol. It is up to 4 t imes faster t han ‘tsunam i’ protoc ol and almost one order of magnit ude faster than the st andard ‘ft p’ prot ocol. Tests were conduct ed repeatedly over several days and at different tim es resulting in slightly different average rates without chan ging the main conclusi on that the UDT protocol is superio r to ‘ftp’ an d ‘tsunam i’ . The diffe rence in data t ransfer rates for different en d points can be explained by different n umber of hop s and diffe rent perf o rmance of sections along the routes. A traceroute comm and from the AUT Blade to the IP address in Mets ähovi gave a rout e of 14 hops. Re-i ssuing t his command repeatedly ove r several months showed that the rout e appears stable, wi thout any chan ges. A high nu mber of hops on the route (14) demonstrates t he comple xity of the pat h and explai ns why data transfers v ia protocols s uch as FTP are not effi cient. A series of additional d ata transfer se ssions was conduc ted in 2010 and 2011 . In August 2010 observations were c onducted of the ESA' s Mars Express spacecraft orbi ting Mars. This data was transmitted t o Metsähovi vi a KAREN. T he data of 24 A ugust whic h totaled 8 6 GB was sent usi ng the UDP pr otocol ‘tsunami’ immediately after the ob servational session. The nex t set of observational data ( 26 August) totaling 187 Gbytes was sent via UDT. Average rates sust ained were 250 and 300 Mbps respectively. Another set o f experim ents was co nducted i n Septem ber 2010 aim ing to test the ‘tsunam i’ protocol for st reaming VLBI data directly from the radi o tele scope receivi ng/recording system (PCEVN) via KAREN to Metsähovi. T his test was an important step towards real-time eVLBI. Initial tests fro m the WRAO to the IBM Blade server at the AUT City Campus show ed that the ‘tsunam i’ rates of 485 Mbps wit h no lost packets was sustai nable. Ho wever, when str eaming data from Wa rkworth to Met sähovi m any thousands o f lost packets occ urred and a su stainable rate of 350 Mbps was achieved. This is sign ificantly lower than the rate of 512M bp s required for the real-time eVLBI streaming of 8bit data to Metsähovi. In June 2010 file transf er tests to the corr elator site at Curtin Un iversity, Perth w ere conducted. Using ‘tsunami’, rate of 300 Mbps was achieved, while UDT was supe rior w ith 400 Mbps. In December 2010 the first eVLBI tests from the Mk5B recorder at the Ob servatory to the server at CSIRO in Australia using UDP were condu cted. The required data rate of 512Mbp s was achieved sustainab ly. In January 201 1 eVLBI tests with CSIRO where undertaken to stream data directly from the Mk5B to the correlator: a rate of 512M bps was achieve d for 8 channels at 2 bit resoluti on. In February 2011 successful real-time eVLB I tests and demonstration were undertaken between the New Zealand 12-m radio telescope and the LBA antenn as in Australia [8]. In addition, in January 2011 connectivity was establishe d with the Geographical Survey Institute of Japan (GSI) and tests were conducted bet ween the blade serve r at the AUT Cit y Campus, A uckland a nd the GSI usi ng “iperf” . Rate of 512 Mbps was ach ieved with some packet loss (0.00021%) , but at slower rates th ere was no packet loss. W hen using UDT for transferring files, the loss of packets at a lo w rate would not be a major factor. Currently the KAREN link between New Zeal and and the US A (LAX) i s relatively lightly loaded with th e maximum capacity of 1 Gbps. Th e tests where conducted to t ransfer a 16 GB Mk5B vector file from the WRAO. A similar rate of about 350 Mbps was achieved with bot h ‘tsunam i’ and UDT proto cols. The next step i s to test the more e fficient UDT prot ocol for di rect data stream ing. However , to make use of UDT instea d of ‘ts unami’ f or streami ng requires t he VLBI software to be modi fied. Som e program ming effo rt is needed as the network cod e is merged within the application code rather than being maintained in a separate library. 4. Conclusion It was dem onstrated that the use of the UDP pr otocol for radio astronom ical data transfe r has the requir ed sustained data transfer rates for eVLBI, but UDT has some advantages over ‘ tsunami’: • UDT is a bett er citizen on t he network l eaving bandwi dth for TCP and other UDP protoc ols, whic h is very important on a shared network suc h as KAREN. • UDT has an application programming interface (API) a llowing easy integration w ith existing or future applications. We investigated the ability to stream collected data vi a UDT and modified the Curtin 16 bit to 2 bit conv ersion program for data stream ing to a rem ote server in a fast and efficient manner ready for the correlato r. This has be en successfully demonstrated between Warkworth a nd the IBM Bl ade Serve r at AUT. Curren tly the JIVE Mk5 code is being modi fied to stream data via U DT to a real-tim e digita l autocorrel ation spec trometer im plement ed on IBM InfoSphere Str eams [9] . We have found KA REN to be a very usefu l tool for tr ansmitting da ta to our internatio nal partner s, and the IRASR will be extending its use over the co ming months. Of futu re benefit to o ur work to stream data real-ti me to the international correlatio n centers would be the ability to reserve bandwidth as a logical pipe within the KAREN bandwidth for the duratio n of an experiment. Acknowledgements This researc h was support ed by the M inistry fo r Econom ic Developm ent's SKA Gra nt. Warkwo rth connecti vity via KAREN wa s made possi ble by the Re mote Sit e Connectivity Fund provide d by the Minist ry of Research Sci ence and Technolog y (MoRST) on behalf o f the New Zealand Governm ent. We are grateful to AUT and KAREN staff, as well as to our VLBI part ners in Australi a, Japan, Finla nd, USA, Germany and the Netherlands for th eir support and assistance. References 1. S. Gulyaev and T. Natusch, “New Ze aland 12-m VLBI Station” , International VLBI Service for Geod esy and Astrometry 2009 Annual Report , NASA /TP–2010-215860 . Eds.: D. Behrend and K. D. Baver, 2010, pp. 138-141. 2. A. K. Tzioumis, S. J. Tingay, B. Stansby, J. E. Reynol ds, C. J. Phillips, S. W. Amy, P. G. Edwards, M. A. Bowen, M. R. Leach, M. J. Kesteven, Y. Chung , J. Stevens, A. R. Forsyth, S. Gulyaev, T. Natusch, J.-P. Macquart, C. Reynolds, R. B. Wayth, H. E. Bignall, A. Hotan, C. Hotan, L. Godfrey, S. Ellings en, J. Dickey, J. Blanch ard, and J. E. J. Lovell, “Evolution of the Pa rsec-Scale Structure of PKS 1934 − 638 Revisited: First Science with the ASKAP and New Zealand Telescopes”, The Astronomical Journal , 140, November 2010, p p. 1506–1510. 3. S. Pogre benko et al, EVN Newsletter , 27, Septem ber 2010 4. L. Petrov, C. Phillips, A. Tzioumis, B. Stansby, C. Re ynolds, H. E. Bignall, S. Gulyaev, T. Natusch, N. Palmer, D. Collett, J. E. Reynolds , S. W. Am y, R. Wayt h, and S. J . Tingay, “First ge odetic obser vations usi ng new VLB I stations ASKAP-2 9 and WARK1 2M”, Public ations of t he Astronomical Society of A ustralia, 2011, submitted . 5. J. Wagner Tsunami UDP Protoc ol [Online]. http://tsunami-udp.sour ceforge.net/ (2010, Sep. 23) 6. UDT UDP-bas ed Data Tr ansfer [Onli ne]. Availa ble: http:/ /udt.source forge.net /index.ht ml (2010, Sep. 23) 7. Yunhong Gu (2005). UDT A High Performance Da ta Transport Protocol (Unpublishe d doctoral dissertation). University of Illinois, Chicago, USA. 8. anzSKA Newsletter 9. M.S Mahmoud (2010). Utilizing a Hetrogeneou s Computing Cluster & IBM InfoSphere Streams (unpublishe d masters dissertation). AUT Univer sity, Auckland, New Zealand.