Integrated monitoring of multi-domain backbone connections -- Operational experience in the LHC optical private network

Inte grated monitoring of multi-domain backbone connections Operational experience in the LHC optical pri v ate netw ork Patricia Marcu, David Schmitz, W olfgang Fritz, Mark Y ampolskiy , W olfgang Hommel Leibniz Supercomputing Centre Boltzmannstr . 1, 85748 Garching, Germany {marcu,schmitz,fritz,yampolskiy ,hommel}@lrz.de ABSTRA CT Novel lar ge scale r esear ch pr ojects often r equir e cooperation between various differ ent pr oject partners that ar e spr ead among the entir e world. The y do not only need huge computing r esour ces, b ut also a r eliable network to operate on. The Lar ge Hadr on Collider (LHC) at CERN is a r epr esentative example for suc h a pr oject. Its e xperiments r esult in a vast amount of data, which is interesting for resear c hers ar ound the world. F or transporting the data fr om CERN to 11 data pr ocessing and stora ge sites, an optical private network (OPN) has been constructed. As the experiment data is highly valuable, LHC defines very high requir ements to the underlying network infrastructur e. In or der to fulfil those r equir ements, the connections have to be managed and monitor ed permanently . In this paper , we present the inte grated monitoring solution developed for the LHCOPN. W e first outline the requir ements and show how they ar e met on the single network layers. After that, we describe, how those single measur ements can be combined into an integr ated view . W e cover design concepts as well as tool implementation highlights. KEYWORDS Multi-Domain-Monitoring; Inte grated Monitoring; V isualisation; GÉANT ; LHCOPN 1 . I N T R O D U C T I O N The significant increase in the av ailability of high-speed research networks has led to the deployment of large-scale distributed computing en vironments that are able to serve a great number of geographically separated users by exchanging huge amounts of data. Direct communi- cation between sites and computing facilities is no w necessary in man y working environments. This results in greatly expanded requirements for high-speed, dedicated networks that cross multiple domains. The European Research Network GÉANT [1] and National Research and Educational Networks (NRENs) in Europe are high-capacity networks, which are based on optical technologies and components that provide wav elength-based services to the research community . A representativ e project is the provisioning of the networking infrastructure for the Large Hadron Collider (LHC) at CERN in Switzerland. Its research experiments produce about 15 petabytes of data per year . Therefore, a multi-domain LHC Optical Pri v ate Network (LHCOPN) was established [2], dedicated to support data exchange. The LHCOPN consists of T ier-0 and T ier-1 centres connected by End-to-End (E2E) links. These E2E links connect organisations (T ier-1 centres) that are located in different countries and cross the shared network infrastructure of different providers tow ards the T ier-0 centre at CERN. One of the most important and difficult issues related to this dedicated network is network management. The monitoring and troubleshooting of optical networks and individual E2E links is challenging. Researchers all over the world are increasingly using dedicated optical paths to create high-speed network connections, and dif ferent groups of users may want to use monitoring applications for specific research purposes. They need access to network measurement data from multiple in v olved network domains, visualise network characteristics and troubleshoot related issues [3]. A quick overvie w and visualisation of the network status is necessary to establish demar- cation points that help distinguish network issues within LHCOPN from those in the sites. The deployment of monitoring tools and the use of common services should provide a unified network information view across all domains [4]. T ypical off-the-shelf solutions do not provide such functionality . This article is structured as follo ws: In Section 2, the requirements in the context of the LHCOPN are described. After that, some important existing approaches in the area of inter- org anizational monitoring are outlined in Section 3. Section 4 explains, how monitoring is done at layer 1 and 2, before monitoring at layer 3 and abov e is added in Section 5. A motiv ation of integrating them and a description of this integration is outlined in Section 6. Section 7 then giv es an ov erview of the implementation and the operational experiences gained within the LHCOPN. The article is concluded in Section 8. 2 . R E Q U I R E M E N T S The monitoring of the LHCOPN, i.e. T ier -0 and T ier -1 centres, and the links between them poses sev eral new challenges: 2.0.0.1. Multi-domain monitoring. The LHCOPN itself is based on resources that are supplied by se veral academic networks such as GÉANT , European NRENs, Internet2, ESnet and Canarie. Therefore, a solution has to be found to collect monitoring data from all these self-administered networks to form a joint view of the resulting network. 2.0.0.2. Monitoring of dif ferent layers. While academic netw orks have been used to mon- itor the network layer , the LHCOPN requires E2E links on the data link layer to be monitored. These are based on heterogeneous technologies, as the different participating networks use dif ferent technologies. E2E links are formed by combining technologies such as SDH/SONET , nati ve Ethernet or Ethernet ov er MPLS, where each domain is dependent on the data that it can retriev e from the network management system of its vendor . 2.0.0.3. Joint view of all metrics. In the visualisation a vie w has to be formed by combining E2E link and IP-related monitoring data and by linking these data in a suitable manner . In doing so, it must be considered that there are also se veral data sources on the IP lev el, in particular the retriev al of SNMP data from routers and the results of activ e measurements. 3 . E X I S T I N G A P P R OAC H E S The issue of multi-domain monitoring is not only a challenge in the context of the LHCOPN, but also in the general operation of networks. In 2004 a collaboration of the GN2/GN3 project with Internet2, ESnet, RNP and others was started to jointly de velop a communication protocol and tool set under the name perfSONAR [5], [6]. This de velopment has become necessary by the limitations of existing tool sets which were tied to single domain monitoring and limited to subsets of metrics that can be monitored. Such limitations apply e.g. to the MonALISA [7] tool set. Apart from being used in the LHCOPN, the perfSONAR tools are used within the networks that participate in the collaboration. As perfSON AR is open source, it is used by other projects like Distributed European Infrastructure for Supercomputing Applications (DEISA) [8] too. There are already se veral tools which can visualise perfSON AR measurement data [9]. One of them is perfsonarUI which can be used for troubleshooting by allo wing a direct interaction with perfSONAR measurements. All principles of the perfSONAR protocol will be taken into account in the customisation for the LHCOPN, especially its multi-domain-monitoring feature. Furthermore, this will be done with respect to the differ ent layers monitoring also dev eloped within the GN2/GN3 projects. The missing requirement for this customisation is the joint view on all metrics . Also, a global ov erview of the LHCOPN w as needed, in which dif ferent layer vie ws coe xist. This customisation p e r f S O N A R C o m m u n i c a t i o n S c h e m a R e t r i e v a l o f d a t a i n X M L f o r m a t N e t w o r k D o m a i n 1 . . n N M S E 2 E M o n i t o r i n g S y s t e m S c r i p t f o r D a t a T r a n s f o r m a t i o n X M L r e g i s t r y f i l e ( c u r r e n t d a t a ) W e b S e r v i c e M P p e r f S O N A R C o m m u n i c a t i o n S c h e m a R e t r i e v a l o f d a t a i n X M L f o r m a t N e t w o r k D o m a i n 1 . . n N M S E 2 E M o n i t o r i n g S y s t e m S c r i p t f o r D a t a T r a n s f o r m a t i o n W e b S e r v i c e M A D B M e a s u r e m e n t A r c h i v e M e a s u r e m e n t A c q u i s i t i o n D a t a T r a n s f o r m a t i o n & A g g r e g a t i o n M A / M P D a t a P o p u l a t i o n D a t a P r o v i s i o n i n g S c e n a r i o u s i n g M e a s u r e m e n t P o i n t ( M P ) G e n e r a l S t e p s S c e n a r i o u s i n g M e a s u r e m e n t A r c h i v e ( M A ) D a t a P r o v i s i o n i n g Figure 1. Functionality of E2EMon MP [11] was achie ved by a dedicated version of the Customer Network Management (CNM) tool [10] (in a browser -based version). 4 . M O N I T O R I N G A T L A Y E R 1 A N D 2 : E 2 E M O N The introduction of hybrid networks and the possibility to deliver E2E links that in v olve multiple domains has led to the need to monitor these links. E2E Links are defined as dedicated optical point-to-point network connections realized at ISO/OSI layers 1 and 2. For the pur- pose of fault detection and localization of such connections, a dedicated tool called E2EMon (E2E Monitoring T ool) [11] has been dev eloped over the recent years. It is used by the E2E Coordination Unit (E2ECU) in GÉANT – an organizational unit established for inter-domain coordination of operational procedures. E2EMon basically consists of two important components—the Monitoring Point (MP) and the central component. Ev ery domain which pro vides a se gment of such an E2E link needs to hav e an E2EMon Measurement Point (MP) in place which retriev es data from the local network management system to provide status information for the link segment. In the following, we will first outline, how the Measurement Points works, continuing with a detailed description of the central component. 4.1. E2EMon measurement point (MP) The Monitoring of circuits crossing dif ferent organizational domains is often a big challenge. It is not quite easy to get all the needed information from all in volved router interf aces and so on. Furthermore, many network operations centres (NOCs) already hav e their o wn local monitoring solution, which the y want to keep. Therefore, an infrastructure and technology independent col- lection of this local monitoring data is needed. The E2EMon MP provides an interface between those provider specific and a provider independent representation of monitoring information. The E2EMon MP is a set of perl scripts. Once installed and deployed in every inv olved domain, it is basically a standalone SO AP server instance. Therefore, it does not rely on En d Po in t B Do main 1 Do main 2 Do main 3 En dP oi nt A Interd oma in Lin k De marc. Po in ts Mea s. P oi nt Mea s. Archi ve E2 E Mon itori ng Sy stem Mea s. P oi nt Use rs, E2 ECU Do mai n L in k Figure 2. Information flow: from domains to E2EMon and E2ECU [11] webservers like tomcat, apache, etc. It does not install itself as a daemon service, but of course can be run as such if it is configured in the operating system. All necessary and required modules are installed together with the service. After that, E2EMon MP is waiting for an XML file to send to the central component. This XML file has to be generated by the particular network monitoring system (NMS) in use, for example, Nagios, Cacti, or others (see Figure 1). Obviously , this XML file has to be updated regularly—otherwise it will not reflect the actual situation. The XML file is based on the schema introduced by the OGF Network Measurements W orking Group (NMWG) [12], [13]. A typical snippet of such an XML file is shown below in Listing 1. Listing 1. Sample E2EMon MP XML file < n m w g : m e s s a g e t y p e = " S e t u p D a t a R e q u e s t " x m l n s : n m w g = " h t t p : / / g g f . o r g / n s / nmw g / b a s e / 2 . 0 / " > < n m w g : m e t a d a t a i d = " m e t a 1 " > < n m w g : e v e n t T y p e > P a t h . S t a t u s < / n m w g : e v e n t T y p e > < / n m w g : m e t a d a t a > < n m w g : d a t a i d = " d a t a 1 " m e t a d a t a I d R e f = " m e t a 1 " / > < / n m w g : m e s s a g e > The XML file itself does not contain specific detailed measurement data, but rather an abstracted view of it. The interfaces and links are categorized as UP , DO WN, DEGRADED, or UNKNO WN, depending on their actual operational state. This mapping has to be provided by the domain’ s monitoring system, respectiv ely by the tool or script that con verts the local monitoring system’ s data to XML. Last but not least, the XML file consists of different sections, each providing information about either a so called monitor ed link or so called demar cation points needed by the central component of the E2E Monitoring System . W e will e xplain these two terms in detail in the next section. 4.2. E2E link Monitoring System (E2EMon) E2EMon relies on the „health” information provided by different NRENs. A term „Monitored Link” is used by E2EMon de velopers and users in order to refer to E2E Link parts, for which monitoring information is provided by NRENs. All monitored links are edge-to-edge connections from a border of one NREN to the border of the same or neighbor NREN. Ends Domain 1 Domain 2 Domain 3 (na t i v e E t her ne t ) (E t her ne t o v er SDH ) (E t her ne t o v er MP LS) Segme n t 1 : I EEE 80 2 . 1 Q V LAN Segme n t 2 : SDH p a t h (usin g GFP - F) Segme n t 2 : MP LS LSP E nd P oin t A E nd P oin t B E2E Link A - B Figure 3. E2E Link, typical built up [14] of Monitored Links are referred to as Demarcation Points (DPs). The Follo wing three types of Monitored Links are supported by E2EMon: • Domain Link is an connection completely realized within a single NREN • InterDomain Link is a whole connection, interconnecting two neighboring NRENs • InterDomain Link Part is the type of connection representing a part of Inter-Domain connection from the perspective of a single NREN The distinguishing between InterDomain Link and InterDomain Link Part is necessary be- cause of typically very restricti ve information and management policies of the in v olved NRENs. In most cases these policies pre vent the measurement of inter -domain connection state. There- fore, „InterDomain Link Part” is used to provide state information from a single NREN’ s perspecti ve. States of two parts can be aggregated to the state of the whole inter-domain connection. Also, the restricti ve information policies cause high abstraction lev els of Monitored Links. One of the challenges, which hav e to be overcome in a multi-domain en vironment is the synchronization of monitoring data. E2EMon does not require clock synchronization in multiple NRENs. Instead, E2EMon polls MPs/MAs of different NRENs periodically (see Figure 2). E2EMon assumes then that data retriev ed from multiple NRENs in the same polling cycle is up to date and synchronized. Currently , a 5 minute polling interval is used. This time interval is treated by E2ECU as fine grained enough. The time needed to collect and to process data from about 30 NRENs is about 1 minute. Because of the independence of NRENs and non-coordinated procurement policies, heteroge- neous hardware and network technologies are used in general to realize E2E Links parts. T ypical are SDH, Ethernet, Ethernet ov er SDH, and MPLS (see Figure 3). Due to high heterogeneity and as an outcome of this a high dif ference of state information, a true technology specific monitoring data cannot be used for multi-domain monitoring. Instead, the following abstracted operational states are used: • UP – the connection is up and running • DEGRADED – the connection is up, but with degraded performance • DO WN – the connection is down and cannot be used at all • UNKNO WN – the state of the connection is unknown Identical aggregation rules hav e been defined for (a) two InterDomain Link Part states to the state of a whole InterDomain Link and (b) all Monitored Links of a particular E2E Link to the whole E2E Link state. The aggregation rules are defined in the way , that the worst state dominates. By implementating such a strategy , v alues hav e been assigned for all supported states (see Figure 4(a)). In the aggre gation rule, the biggest state-weight of in volv ed Monitored Link is used as a weight of the whole E2E Link. If one or more Monitored Link states are UNKNO WN, the GUI shows a warning independent of the aggregated state. Operat ional St ate Value Weight Unknown 0 Up 1 Degraded 2 Down 3 (a) Operation States Administr ativ e S tat e Value Description Weight Unknown Do mai n co uld not acqu i re i nformatio n abou t admi ni stra tiv e state 0 NormalOperation No admi nis t r ativ e work is per fo rmed 1 Maintenance Planned m ain tenan ce a ctivi ty in pr ogr e ss 2 TroubleShooting Trouble shoot ing is in prog r ess 3 UnderRepair Re p air p roce ss i s in pro gres s 4 (b) Administrative States Figure 4. E2EMon States [11] In addition to the operational state, E2EMon supports an administrativ e state of monitored and E2E links. The supported states, their weight and short description are listed in Figure 4(b). The rule to aggregate the administrativ e state is identical to the one used for the operational state. The administrativ e state reflects the management processes performed by the domains. This state is used by E2EMon in order to, e.g., pre vent a new alarm from being raised if the Monitored Link goes DOWN during a planned maintenance. In practice, some of the described features of E2EMon are not used. As there is no strict definition of semantics for DEGRADED state, NRENs generally omit to report it ev en if the degraded performance is recognized. Another unused feature is the possibility that an NREN reports the whole state of an Inter-Domain connection. In order to do this, an access to the infrastructure of a neighbor NREN is needed. But as this collides with the restricti ve security policies of some NRENs, only reports of Domain Link Parts are used currently . Finally , the administrati ve state is used by fe w NRENs only . The coupling of NREN-specific management tools and the export of the data for E2EMon is difficult. The state of E2E Links is presented in the E2EMon GUI. All E2E Links are sho wn in a semi-graphical view , which provides information about the Monitored Links and their order in the link. E2EMon does not require information about order and arrangement of Monitored Links in advance. Instead, E2EMon computes this information implicitly as follo ws: • For e very Monitored Link the globally unique E2E Link ID is provided, which allo ws to group information provided by different NRENs. • Every Monitored Link is further specified by globally unique IDs of its Demarcation Points. Step 1 DFN SU RFn et DFN - LR Z DFN - MUE DF N - MUE SUR Fne t - MUE SUR Fne t - MUE SUR Fne t - SARA DFN - LR Z DFN - MUE DFN - MUE SUR Fne t - MUE SUR Fne t - MUE SUR Fne t - SARA DFN - LR Z DFN - MUE SUR Fne t - MUE SUR Fne t - SARA Step 2 Figure 5. Reconstruction of an E2E Link [11] Based on this information, dif ferent Monitored Links are „stitched together” at the DPs with the same IDs (see Figure 5). In some cases, the reconstruction of the whole E2E Link structure might not be possible. This can happen, e.g., if some MPs/MAs can not be reached due to fire wall restrictions. It is also possible that information about monitored links is provided using wrong IDs for DPs. Such situations sometimes occur if new E2E Links are setup. In those cases, E2EMon assembles as many pieces of an E2E Link as possible and displays them with the icons for gaps between contiguous sections (see Figure 6). S t a rt page H e l p pa ge S y s t em E rro rs E2ECU vi ew E 2E Lin ks M on. L i nk s P ro bl em E 2 E Li nk s P ro bl em M on . Li nk s D o m ain v ie w CA N A RI E C E RN ( ?) C ESN E T D EI SA D FN E SN ET FER MI ( ?) GA RR G EAN T 2 I N 2 P3 I N TE R N E T2 N E T HE RLI G HT PS NC RE D IR IS R EN AT ER R IPN ( ?) SA RA S W IT CH U L A KBI M ( ?) U S LHCNET Pr o j ec t vi ew C B FA D EI SA EX PRE S G LIF GRI D5K I G T MD L H C O PN LHC T1T 2 PHO SP HORUS T I E R2S U S LHCNET S t at us of E2E Link FER M I-SA R A - LHC O P N- 00 1 Ti m e of St a t e A ggre gra t i on: 2 010 -09-09 , 10: 1 6: 36 M E S T ( C y c l e t i m e: 60 s . ) O p er at i o n al S t at e: Up A d m i n i s t r at i v e S t at e: N o r m al O p er . E rror: E2E Li n k is not c o nt i gu ous (E n d P oi nt m i s s i ng o r gap foun d) W a rni ng: O pe rat i on al s t a t e i s not k n own f o r al l i nv ol v ed l i nk s W a rni ng: A d m i ni s t rat i v e s t at e i s not k nown for al l i nv ol v e d l i n k s D oma in FE RMI ( ?) FE RMI ( ?) USL HCNE T Link Struc ture Ty pe E nd P o i n t ID P ar t . In fo ID P a rt . Inf o De m arc Gap De marc Dom ai n Li n k Dema rc Lo c a l N a me FE R MI- S ARA S t arl i ght _ F NA L - i f28 S t arli ght _F NA L- if28 FE R MI- ES NET - US L H CNET- A MS FER MI - SARA - LH C OPN - 001-CHI-A MS U S L HCNE T- CHI St at e O p er . - Up Up - - - Up - Sta te Admin. - N orma l Ope r. Norma l Oper. - - - Norma l O pe r. - Times t a mp - 201 0-09 -09 T 08 : 15: 0 2. 0- 6: 00 2 010 -09-09 T08: 1 5: 02. 0 - 6: 00 - - - 201 0-09-0 9T 10 : 11: 1 7Z - P ag e ge ne r at e d at 20 10 - 09 - 09 , 1 0: 17 : 02 M E ST 09.09.2010 E2E Li nk Mo nito ring Sys tem … lr z-m ue nc he n. de/ … /G2_E 2E _in dex_… 1/1 Click to buy NOW! P D F - X C H A N G E w w w . d o c u - t r a c k . c o m Click to buy NOW! P D F - X C H A N G E w w w . d o c u - t r a c k . c o m Figure 6. GUI representation of an E2E Link [15] E2EMon distinguishes between productiv e and not yet productive E2E Links. For productiv e E2E Links, monthly and weekly av ailability statistics are collected. These statistics contain not only UP and DO WN time of a particular E2E Link, but also the so called „Uncertain-Time”. If the E2E Link structure could not be fully reconstructed, the state of the link is calculated based on the information of known monitoring links. At the same time, this state is considered as „uncertain” as it can be influenced by states of unkno wn monitored links. If an E2E Link is in uncertain state, the polling period is added to the link’ s uncertain time counter . The a vailability of an E2E Link is computed as certain Up-T ime di vided by the ov erall monitoring time of the link. Statistical information can be accessed via the GUI and also saved as a CSV (comma separated value) export file (see Figure 7). S t a rt page H e l p pa ge S y s t em E rro rs E2ECU vi ew E 2E Lin ks M on. L i nk s P ro bl em E 2 E Li nk s P ro bl em M on . Li nk s D o m ain v ie w CA N A RI E C E RN ( ?) C ESN E T D EI SA D FN E SN ET FER MI ( ?) GA RR G EAN T 2 I N 2 P3 I N TE R N E T2 N E T HE RLI G HT PS NC RE D IR IS R EN AT ER R IPN ( ?) SA RA S W IT CH U L A KBI M ( ?) U S LHCNET Pr o j ec t vi ew D EI SA I G T MD L H C O PN LHC T1T 2 A v aila b ilit y S t a t is t ic s C u rrent W ee k C ur r e n t M on th A l l M on t hs A l ar ms A la r m s by I D D EI SA- 0 0 1 F R A _ G AR G A R _ F RA D E ISA BSC - F R A- D EI SA- 0 0 1 DEI S A- F R A _ M AD D EI SA- M A D_FRA G E A N T2 D E ISA 99. 93 121 39 1 2131 0 0 8 P ro je c t : IGT M D C SV Exp o r t E 2 E L i nk I D E ndp oi nt A E ndpoi nt B Moni t ore d D om a i ns A v a ila b ilit y ( %) Moni t ore d T ime ( m i n) U p- T ime ( m i n) D e gra de d- T ime ( m i n) D ow n- T ime ( m i n) U nc e rt a i n- T ime ( m i n) F E R M I- I N 2 P 3- I G T M D -0 02 F E RMI- I G T MD I N2P 3- I GT M D2 I NT E RNET2 G EAN T 2 R E N AT ER FE R MI E SN ET I N 2 P3 9 8. 49 1213 9 1195 6 0 10 173 F E R M I- I N 2 P 3- I G T M D -0 01 F E RMI- I G T MD I N2P 3- I GT M D1 I NT E RNET2 G EAN T 2 R E N AT ER FE R MI E SN ET I N 2 P3 9 8. 57 1213 9 1196 6 0 0 173 P r oj ec t : L HCO P N C SV Exp o r t E 2E L i nk ID E ndpoi nt A E ndpoint B Moni tor e d Doma i ns A vaila bilit y ( %) M onitore d Time (min) Up- Time (min) D e gra de d- Time (min) Dow n- Time (m in) Unc e rt a i n- Time (min) CE R N- T RIUM F- LHCOPN- 0 02 CERN-T 0 C AN AR I E- V N C V2 OM E1 I NT E RNET2 G EAN T 2 NE T HE R L IGHT C AN AR IE 99 . 90 1 2139 121 27 0 0 12 CE R N- F E R M I- LHCOPN- 0 01 CERN-T 0 F E R M I - T1 U SLH C NET FE RMI E SN ET 99 . 98 1 2139 121 37 0 0 2 CE R N- SAR A- LHCOPN- 0 01 SA R A- GRI DR1 C E RN-T 0 G EAN T 2 NE T HE R L IGHT SA RA 98 . 43 1 2139 119 49 0 148 42 CE R N- F E R M I- LHCOPN- 0 02 CERN-T 0 F E R M I - T1 U SLH C NET FE RMI E SN ET 99 . 98 1 2139 121 37 0 0 2 CE R N- CNAF - LHCO PN- 0 01 GA RR- C N AF C E RN-T 0 G EAN T 2 G ARR 99 . 90 1 2139 121 27 0 0 12 09.09.2010 E2E Li nk Mo nito ring Sys tem … lr z-m ue nc he n. de/ … /G2_E 2E _in dex_… 1/1 Click to buy NOW! P D F - X C H A N G E w w w . d o c u - t r a c k . c o m Click to buy NOW! P D F - X C H A N G E w w w . d o c u - t r a c k . c o m Figure 7. Monthly statistics of productive E2E Links [15] E2EMon is a tool integrated in manual operational processes. Alongside with the GUI, sev eral interaction interf aces ha ve been implemented. In case an E2E Link state changes to DEGRADED or DOWN, E2EMon automatically sends email notifications to E2ECU and in volved NRENs. In order to simplify the inte gration of E2EMon with other management tools lik e Nagios, SNMP traps are generated every time an E2E Link state changes. The state of all producti ve E2E Links is exported using an XML file. This XML export is updated e very polling interv al and used by LHCOPN W eathermap for integrating layer 1 and 2 states with other measurements. 5 . M O N I T O R I N G A T L A Y E R 3 A N D A B OV E : H A D E S A N D B W C T L For IP-lev el monitoring, the HADES and BWCTL tools are of interest, as they provide rele vant metrics and are integrated into the perfSON AR frame work. Therefore, they were easily integrated as a part of the LHCOPN overall management solution. 5.1. HADES HADES (Hades Activ e Delay Evaluation System) [16] uses dedicated hardware boxes to perform activ e tests in the network to measure delay , jitter , packet loss and traceroute (with respect to IPPM recommendations [17]). For precise timing, GPS antennas are installed in addition to the hardware boxes. Networking T ime Protocol (NTP) can also be used but with less precision. The HADES [16] boxes hav e been deployed at the Tier -0 and Tier -1 LHCOPN locations to provide QoS measurements. The HADES topology is made up of the abstract nodes, so it can easily link them to E2E measurements and directed (uni-directional) HADES links between them. A HADES link is determined by its source and targeted abstract nodes. As HADES links correspond to pairs of abstract nodes, they are identified by ordered pairs of abstract locations. HADES measurements are run as a full mesh between all nodes. T o pre vent users from being ov erloaded with too much data, the visualisation in the integrated view (the so-called W eathermap , see Section 6) is limited to measurements that relate to paths where E2E links exist. The metrics on the HADES layer are IP performance metrics (IPPM) computed for each HADES link (one way delay [18], IP delay variation (jitter) [19] and packet loss [20]) as well as the hop list/count metric. As the links are directed between two different HADES end points A and B, the metrics exists for A → B and B → A . All these metrics hav e a time resolution of 5 minutes. The HADES metrics are stored in a HADES Measurement Archive (based on the SQL MA), which is used to store and publish historical monitoring data produced by the HADES Measurement Points. 5.2. BWCTL Similar to HADES, BWCTL (Bandwidth T est Controller) [21] verifies av ailable bandwidth from each endpoint to other points to identify throughput problems. In the LHCOPN, BWCTL nodes are included within the HADES boxes by using a second interface card. Each BWCTL end point address is associated with an abstract node. This is a 1:1 mapping but the IDs of the BWCTL end point and of the abstract node are not the same (BWCTL IP addresses vs. location names). On this layer , the needed metrics are minimum, medium and maximum BWCTL throughput (stored in the SQL MAs). As the BWCTL links are directed between two dif ferent BWCTL end points A and B, these BWCTL metrics exist for both directions A → B and B → A . 6 . I N T E G R A T E D V I E W : L H C O P N W E A T H E R M A P Network operators rely on monitoring information. Different layers provide different informa- tion of the monitored links. In order to foster the operational procedures, an inte grated view of the monitoring information is needed. T o provide such an integrated view , all the measurements important to the LHCOPN have to be identified and a mechanism has to be defined how they can be combined. 6.1. Measurements in LHCOPN T o understand the metrics displayed in the LHCOPN W eathermap, it is necessary to know ho w the measurements are carried out. The deployment of the perfSON AR measurement tools at each T ier-1-centre is therefore shown in Figure 8. Figure 8. T ier-1 site configuration First of all, data is collected for the E2E links (see Section 4). The links start at the T ier-0 centre or at one of the T ier-1 centres (backup links), end at one of the T ier-1 centres and typically cross sev eral administrativ e domains (e.g. GÉANT , European NRENS, Internet2 or ESnet). The status of each link is then calculated based on the NMS data from each domain in volved. In addition to this E2Emon MP , three measurement servers are located at each centre: • HADES/BWCTL MP: The first server is the HADES box, which conducts one-way delay [18], IP delay variation (jitter) [19], packet loss [20], and traceroute tests with any other HADES box in the LHCOPN e very minute. It is connected to a GPS antenna for precise timing. In addition, the HADES box hosts a BWCTL MP which is used for throughputs with the Tier -0 centre e very 8 hours. The BWCTL MP is hosted on a dif ferent interface to av oid interference with HADES measurements. • T elnet/SSH MP: The second server is used for the T elnet/SSH MP , a tool that allo ws con- figuration data to be retriev ed from the routers. It is only mentioned here for completeness, but does not carry out any regular measurements. • The last server is used to host an RRD MA to collect utilisation, interface errors and output drop data related to the router located at the T ier-1 centre. 6.2. Integrating the monitoring data The measurements that are carried out hav e to be displayed in a suitable manner , which means in this case that a trade-of f between correct display and usability has to be made. For example, HADES measurements are not directly located on the routers, so that delay data is not exactly measured at the location of utilisation measurements. Events on the short link between router and HADES box can lead to wrong interpretations. Even more difficult considerations have to be made for E2E link status data and its relation to IP metrics. By default the IP data in the network uses the direct way via an E2E link. Ho wev er , if the E2E link fails (including the optical protection), then the IP protection performs a rerouting. Although IP packets are still transferred, they take another physical route. Therefore, it is necessary to clearly distinguish between optical and IP le vel. For this reason, a data model is introduced in the following that cov ers all topology infor- mation per network layer and all necessary topology mapping information for the LHCOPN W eathermap. W ith respect to the requirements stated in Section 2, the operators of the LHCOPN need a global view on their network. Other essential aspects are layer-related, location-related and metric-related vie ws on the LHCOPN. An E2E view is needed to check the a vailability of the E2E links in volved. Therefore, as described in the former sections different layers (topologies) hav e been defined: E2E link, HADES and BWCTL. Information about the IP links between two different IP interfaces is needed to determine the status of the links between two IP interfaces within the LHCOPN (these are VLANS in their terminology). The router topology consists of the abstract nodes, which here relate to IP interfaces and IP links (pairs of IP interfaces). One IP link corresponds to a VLAN in the LHCOPN terminology . The current assumption is that one abstract link is associated with one IP interface pair only . This means that, if one abstract link has two or more E2E links, they both contribute (in an aggregated manner) to the same IP link (back-up link or b undle of links). Also, one E2E link can contribute to a single VLAN only . The metrics used on the router topology are utilisation, input errors and output drops for each end point of an IP link. These metrics ha ve a time resolution of 5 minutes. T ypically , the y are retrie ved via SNMP from routers and stored in so called RRD MAs (Round Robin Database Measurement Archiv e) or SQL MAs (Structured Query Language Measurement Archiv e). These are tools that provide archived measurement data. 7 . O P E R A T I O N A L E X P E R I E N C E A N D I M P L E M E N T A T I O N H I G H L I G H T S T o satisfy the LHCOPN requirement for multi-domain monitoring accessed through a global vie w , a layer based on the E2E link has been specified to form the main vie w . This layer gi ves an ov erview of the whole LHCOPN, on the dedicated E2E links in volv ed in the LHCOPN. For each link that is displayed in the topology two kinds of abstractions can be in volv ed. A link represented here can be an E2E Link or it can be an E2E link plus another E2E link which serves as optical (1+1) protection. The other kind of abstraction that is in volv ed, is that for each E2E link the status is deri ved from data retrieved from multiple NMS. In the following a detailed description of the E2E Link topology , whose representation in the LHCOPN W eathermap is sho wn in Figure 9, is giv en. The topology consists of abstract nodes and abstract links . Abstract nodes represent the Tier - 0 and T ier-1 LHCOPN locations and are named accordingly . They abstract the exact location where measurements are conducted in order to allo w easy linking of this topology to the other topologies. The abstract links are non-directed (i.e. bidirectional) links between the abstract nodes. The metric used for this abstract layer is the aggr e gated status for each abstract link. It is computed ev ery 5 minutes from the E2EMon status of all associated E2E links. For a single E2E link the status is retriev ed in the E2EMon system by polling all E2EMon MPs e very 5 minutes. Figure 9. A view on the E2E Link T opology T ab in the LHCOPN W eathermap tool 7.1. Data retrie val, filtering and integration The data retriev al is concerned with the fetching and updating of topology information, topology mapping information, metric mapping information (see Sections 4 and 5), as well as the actual metric data fetching. The measurements in LHCOPN as well as metric data fetching were outlined in Section 6.1. The abstract topology and its associated E2E links have to be imported from a structured, static configuration file provided by LHCOPN users or , in the future, from an online config- uration file. The E2E Link topology is fetched from the E2EMon export interface together with their indi vidual E2E link states and ha ve to match the ones (associated with abstract links) specified abov e. The HADES topology is imported from metadata of the LHCOPN HADES MA. The topology mapping (abstract node 1:1 HADES node) is tri vial, and has to be altered to show links that are interesting to the W eathermap (links that correspond to an abstract link). The BWCTL topology is imported from metadata in the LHCOPN BWCTL SQL MA. The mapping between BWCTL nodes (BWCTL IP addresses) and abstract nodes is statically con- figured. Potential LHCOPN IP interface address pairs are imported from the metadata of v arious LHCOPN RRD MAs and then need to be altered according to the abstract link to IP interface address pair mappings specified above. 7.2. V isualisation T o meet the requirements of LHCOPN users, a visualisation consisting of three tabs has been designed: Overview Map T ab , Metric T ab and E2E Link T ab . (a) The link status tab (b) The node status tab Figure 10. The link and status tabs 7.2.1. The Overview Map T ab In this tab (see Figure 9) a map consisting of abstract nodes and abstract links (described in Section 5) is sho wn. This ov erview map indicates the current status using four colours: RED, YELLO W , GREEN and BLUE for the current abstract link status DO WN, W ARNING, UP and UNKNOWN (defined in Section 5). In addition to that, the fifth status (MA GENT A) does not represent a value of the metric’ s aggregated status, but instead indicates that there is a serious mismatch in the topology mapping concerning the abstract link; namely that all associated E2E links (from the point of vie w of the abstract topology) are unknown to the E2E topology (from the point of view of the E2E topology). This status is called topology unknown and indicates that no aggregated status for the abstract link could be computed. The content of the other two tabs (Metric tab and E2E Link tab) is shown when clicking either on an abstract link or an abstract node in the map. So by clicking and choosing the selected abstract element, data corresponding to this abstract link or node is loaded in the metric tab and E2E link tab . 7.2.2. The Metric T ab The metric tab shows statistical graphs of metrics associated to particular abstract links. If an abstract link in the overvie w map is selected, data for this specific link is shown. If a Tier -1 abstract node is selected, the abstract link from the T ier-0 abstract node (CERN) to this selected Tier -1 abstract node location is selected. If the T ier-0 abstract node (CERN) is selected, data for all abstract links from CERN to any T ier-1 is displayed. In the metric tab, 24-hour metric graphs of various metrics of the network layers (see Sections 4 and 5) are presented for the chosen abstract link. The list of visualised metrics is different depending whether the selected abstract element is a node or a link. Metrics for an abstract link: Selecting an abstract link in the overvie w map displays the follo wing metrics for this link (see Figure 10(a)) in the Metric tab: • The graph of the E2E aggregated status associated with the abstract link itself. This is based on the data model in Section 7.1 and is visualised in the pre viously mentioned status colours. • The RRD MA metrics graphs (see Section 6.2 for the single IP link associated with the abstract link. These are visualised for both IP interfaces at both end points of the IP link. Figure 11. The E2EMon status tab All the metrics are measured and updated ev ery 5 minutes. Metrics for an abstract node: Selecting an abstract node in the overvie w map (all T ier-0 to T ier-1 abstract links related to the selected abstract node) displays statistic graphs of the follo wing metrics for the chosen abstract links (see Figure 10(b)): • The Hop count metric graph is divided into differently coloured areas, indicating different routes. • The HADES metrics are visualised as scatter plot graphs (values are dots), each with a 5 minute time resolution. For one way delay and jitter the minimum, medium and maximum is needed. • BWCTL metric graphs are visualising the minimum, medium, and maximum BWCTL throughput. 7.2.3. The E2E Link T ab The metric tab specified in the previous Section sho ws metrics on dif ferent (netw ork) layers in a more end-to-end like fashion between the T ier-0/T ier-1 locations. In addition to this, the E2E link tab (see Figure 11) presents a section status view for the focused abstract link. This is done by wrapping the HTML page for the E2EMon section status for each E2E link associated to the focused abstract link in the map overvie w tab . If the selected element in the map overvie w tab is an abstract link, the E2EMon segment status is shown for all E2E links associated to this. If the selected element in the map overvie w tab is an abstract Tier -1 node, the E2EMon segment status is shown for all E2E links associated with this focused abstract link. 7.3. Client and Access Point The client is implemented as a dynamic HTML page with some jav a script code used for the tabbing interface. T o speed up the access, some graphical parts of the content are created and cached in adv ance: • The current 24-hour statistic plot of any network element necessary as specified in Sections 4 and 5 are usually updated on a 5 minutes basis. • The overvie w map which includes the link status colour is updated ev ery 5 minutes. • Internal to the HTML dynamic generation scripts, additional data base content caching is performed to speed up access further . 8 . C O N C L U S I O N A N D F U T U R E W O R K In this article, the monitoring of the LHCOPN has been explained with a focus on the LHCOPN W eathermap. The support structure is ready to fulfil its needs and has proven its usefulness in day-to-day operations since the LHC experiments are running, and large amounts of data are actually transferred via the network. Besides continuous improvements to the already existing tools, an alarm tool is currently under dev elopment. It is designed to be quite flexible in terms of alarm generation, to be suitable for different user needs. The perfSON AR services used for the LHCOPN and the W eathermap provide a good basis for the future large scale projects in Europe. A collection of such projects can be found in the roadmap of the European Strategy Forum on Research Infrastructures (ESFRI) [17]. For the W eathermap, this means that different ways of customisation to meet the needs of other projects are going to be in vestigated. For projects that want to use dynamic circuits, the perfSON AR group is already inv estigating suitable monitoring methods. A C K N OW L E D G E M E N T S The authors would like to thank their colleagues at the Leibniz Supercomputing Centre of the Bavar - ian Academy of Sciences and Humanities (see http://www .lrz.de/) for helpful discussions and valuable comments about this paper . The authors wish to thank the members of the Munich Network Management T eam (MNM-T eam) for helpful discussions and valuable comments on previous versions of this paper . The MNM T eam directed by Prof. Dr . Dieter Kranzlmüller and Prof. Dr . Heinz-Gerd He gering is a group of researchers at Ludwig-Maximilians-Univ ersität München, T echnische Universität München, the Univ ersity of the Federal Armed Forces and the Leibniz Supercomputing Centre of the Ba varian Academy of Sciences and Humanities. See http://www .mnm-team.org/. R E F E R E N C E S [1] GÉANT, “Géant Homepage, ” http://www .geant.net/, 2010. [2] E.-J. Bos, E. Martelli, and P . Moroni, “LHC T ier-0 to T ier-1 High-Lev el Network Architecture, ” CERN, T ech. Rep., 2005. [3] J. W . Boote, E. L. Boyd, J. Durand, A. Hanemann, L. Kudarimoti, R. Lapacz, N. Simar , and S. T rocha, “T owards Multi–Domain Monitoring for the European Research Networks, ” in Selected P apers fr om the TEREN A Networking Confer ence . TEREN A, Oct. 2005. [Online]. A vailable: http: //www .terena.org/publications/tnc2005- proceedings/ [4] M. K. Hamm and M. Y ampolskiy , “Management of Multidomain End–to–End Links. A Federated Approach for the P an–European Research Network 2, ” in Moving fr om Bits to Business V alue: Proceedings of the 2007 Inte grated Management Symposium . München, German y: IFIP/IEEE, May 2007, pp. 189–198. [5] A. Hanemann, J. Boote, E. Boyd, J. Durand, L. Kudarimoti, R. Lapacz, N. Simar , M. Swany , S. Trocha, and J. Zurawski, “Perfsonar: A service-oriented architecture for multi-domain network monitoring, ” in Proceedings of the 3rd International Confer ence on Service-Oriented Computing (ICSOC 2005) . Amsterdam, The Netherlands: ACM, December 2005, pp. 241–254. [6] “perfSONAR project, ” http://www .perfSON AR.net. [7] “MONitoring Agents using a Large Integrated Services Architecture (MonALISA), ” http://monalisa.caltech.edu/, California Institute of T echnology . [8] “Distributed European Infrastructure for Supercomputing Applications, ” http://www .deisa.eu/, 2010. [9] A. Hanemann, V . Jeliazkov , O. Kvittem, L. Marta, J. Metzger, and I. V elimirovic, “Complementary V isualization of perfSONAR Network Performance Measurements, ” in Pr oceedings of the International Confer ence on Internet Surveillance and Pr otection (ICISP) , v ol. 2006. Cap Esterel, France: IARIA/IEEE, Aug. 2006. [10] “CNM for GN3 project homepage, ” http://sonar1.munich.cnm.dfn.de/. [11] M. Hamm and M. Y ampolskiy , “E2E Link Monitoring: System Design and Documentation, ” GN2 Project, T ech. Rep., 2008. [Online]. A vailable: https://wiki.man.poznan.pl/perfsonar- mdm/images/perfsonar- mdm/1/12/ GN2- JRA4- 06- 010v240.pdf [12] “Network Measurements W orking Group (NMWG), ” http://www-didc.lbl.gov/NMWG, Global Grid F orum. [13] M. Swan y , J. Zurawski, and D. Gunter , “NMWG Schema Developers Guide, ” http://www .cis.udel.edu/ swany/nmwg/devguide-05-dec-04.pdf, Dec. 2004. [14] M. Y ampolskiy , “Maßnahmen zur Sicherung von E2E-QoS bei V erketteten Dienste, ” Dissertation, Ludwig– Maximilians–Univ ersität München, Oct. 2009. [15] “E2E Link Monitoring System, De veloper installation, ” http://cnmde v .lrz- muenchen.de/e2e/lhc/, 2010. [16] “HADES, ” http://www .win-labor .dfn.de/cgi-bin/hades/selectnew .pl?config=, WiN - Labor, RRZE, Erlangen. [17] V . Paxson, G. Almes, J. Mahdavi, and M. Mathis, “Frame work for IP Performance Metrics, ” USA, T ech. Rep., 1998. [18] G. Almes, S. Kalidindi, and M. Zekauskas, “A One-way Delay Metric for IPPM, ” USA, T ech. Rep., 1999. [19] C. Demichelis and P . Chimento, “IP Packet Delay V ariation Metric for IP Performance Metrics (IPPM), ” USA, T ech. Rep., 2002. [20] G. Almes, S. Kalidindi, and M. Zekauskas, “A One-w ay Packet Loss Metric for IPPM, ” USA, T ech. Rep., 1999. [21] “Bandwidth T est Controller (BWCTL). ” [Online]. A vailable: http://www .internet2.edu/performance/bwctl/ bwctl.man.html A utors Patricia Marcu receiv ed diploma in Computer Science in 2006 at the Ludwig Maximilians Universität (LMU) München. In 2007 she joined the MNM T eam at Leibniz Supercomputing Centre in Garching/Munich as a research assistant and pursues her Ph.D. degree in Computer Sci- ence. She is curently working on the further dev elopment Customer Net- work Managemnt (CNM) tool and on the visualisation of the LHCOPN within the european Géant projekt. Her research interest focuses on inter- oganisational fault management and IT Service Management. Da vid Schmitz reciev ed the Ph.D. in computer science from Ludwig Maximilians Universitä t (LMU) München in 2008. Since 2002 he is work- ing at the Leibniz Supercomputing Center (LRZ) in Germany at the begin- ing for the German Research Netwok and later for the European project Géant. He designed and developed the Customer Network Managemnt (CNM) tool. Curently he is working on the further de velopment CNM tool and on the visualisation of the LHCOPN within the european Géant projekt. W olfgang Fritz receiv ed diploma in Information T echnology in 2009 from the department of Electrical Engineering at T echnische Universität München. In late 2009, he joined the Leibniz Supercomputing Centre in Garching/Munich as a research assistant. His current research focusses on multi-domain circuit management and monitoring, especially within the European Géant project. Of special interest are not only technical, but also organizational approaches. Mark Y ampolskiy has a Ph.D. in computer science from Ludwig Max- imilians Univ ersität (LMU) München. Since 2006 he is working at the Leibniz Supercomputing Center (LRZ) in German y for the European project Géant. He is also a member of the MNM team. His current research focuses on QoS assurance for point-to-point network connection and on the cryptographical aspects of IT security . W olfgang Hommel has a Ph.D. in computer science from Ludwig Maximilians Uni versität (LMU) München, and heads the network services planning group at the Leibniz Supercomputing Centre (LRZ) in Germany . His current research focuses on IT security and priv acy management in large distributed systems, including identity federations and Grids. Em- phasis is put on a holistic perspectiv e, i.e. the problems and solutions are analyzed from the design phase through software engineering, deployment in heterogeneous infrastructures, and during the operation and change phases according to IT service management process framew orks, such as ISO/IEC 20000-1.