An application of incomplete pairwise comparison matrices for ranking top tennis players

An application of incomplete pairwise comparison matrices for ranking top tennis pla y ers S´ andor Boz´ oki ∗ – L´ aszl´ o Csat´ o † – J´ ozsef T emesi ‡ No vem b er 3, 2016 Abstract P airwise comparison is an imp ortant to ol in multi-attribute decision making. P airwise comparison matrices (PCM) hav e been applied for ranking criteria and for scoring alterna- tiv es according to a given criterion. Our pap er presen ts a sp ecial application of incomplete PCMs: ranking of professional tennis play ers based on their results against each other. The selected 25 pla yers hav e b een on the top of the A TP rankings for a shorter or longer p erio d in the last 40 y ears. Some of them ha ve never met on the court. One of the aims of the pap er is to pro vide ranking of the selected pla yers, ho wev er, the analysis of incomplete pairwise comparison matrices is also in the focus. The eigenv ector metho d and the logarithmic least squares metho d were used to calculate w eights from incomplete PCMs. In our results the top three play ers of four decades were Nadal, F ederer and Sampras. Some questions hav e b een raised on the prop erties of incomplete PCMs and remains op en for further inv estigation. Keywor ds : decision supp ort, incomplete pairwise comparison matrix, ranking 1 In tro duction A well-kno wn application field of pairwise comparison matrices (PCMs) is multi-attribute deci- sion making (MADM). The v alues of pairwise comparisons are applied for ranking of criteria or for scoring alternatives to a giv en criterion. This pap er will use pairwise comparison v alues for ranking of tennis play ers based on their results against each other. Our aim is to make a ’historical’ comparison of top tennis play ers of the last 40 years. The ranking idea is how the play ers p erformed against each other in a pairwise manner in the long run. W e ha ve collected the results of 25 play ers who hav e b een on the top of the A TP ranking lists for a shorter or a longer perio d. 1 ∗ Institute for Computer Science and Control, Hungarian Academ y of Sciences (MT A SZT AKI), Lab oratory on Engineering and Management Intelligence, Research Group of Op erations Research and Decision Systems and Department of Op erations Researc h and Actuarial Sciences, Corvinus University of Budap est, Hungary e-mail: b ozoki.sandor@sztaki.mta.h u † Department of Operations Research and Actuarial Sciences, Corvinus Universit y of Budap est and MT A-BCE ”Lend¨ ulet” Strategic Interactions Research Group, Hungary e-mail: laszlo.csato@uni-corvinus.h u ‡ Department of Op erations Researc h and Actuarial Sciences, Corvinus University of Budap est, Hungary e-mail: jozsef.temesi@uni-corvinus.h u 1 One can ask, why not an all-time ranking? The answer is simple: our data collection used the official A TP website. The A TP database contained reliable and complete data from 1973 (see at http://www.atpworldtour. com/Players/Head- To- Head.aspx ). 1 There could be several reasons wh y some elemen ts of a P C M are missing. It can happen that decision makers do not hav e time to make all comparisons, or they are not able to make some of the comparisons. Some data could ha ve lost, but it is also p ossible that the comparison was not p ossible. In our case the reason of missing elemen ts is ob vious: w e are not able to compare those play ers directly who hav e never pla yed against each other. Professional tennis is v ery popular around the world. The professional tennis associations (A TP , WT A) hav e b een collecting data ab out the tournaments and the play ers. There is a freely a v ailable database ab out the results of the top tennis play ers including data from 1973. That ga ve the possibility to construct the pairwise comparison matrices of those pla y ers who ha ve b een leading the A TP ranking for a perio d of any length. Applying one of the estimation metho ds for generating a w eight vector w e can pro duce an order of the play ers: a ranking. That approach migh t b e highly disputable among tennis fans, of course, but we hav e to note that other ranking ideas are also based on consensus or tradition, and there is no unique answer to the question ’Who is the b est?’. The existing A TP rankings, for instance, give points to the pla y ers for certain p erio ds accord- ing to the imp ortance of the A TP tournaments (based on the prize money) using simple rules for correcting the impacts of some biasing conditions. The media and most of the experts consider #1 of the A TP-ranking as the ’b est’ tennis pla yer. Our approach is also ranking-orien ted, but we will not use this term, the emphasis will b e put on the excellence of pla yers with higher positions relative to those who ha ve lo wer ranking positions. Ranking of pla yers will b e done according to the weigh ts, and the play er with the highest weigh t can b e regarded as the ’b est’, ho wev er, this term is restricted to our sample of play ers and v aries as different ranking lists are generated. In recen t y ears some pap ers ha ve attempted to rank professional tennis play ers with the use of w ell-founded metho ds. Radicchi ( 2011 ) considered all matches play ed b etw een 1968 and 2010 to construct a weigh ted and directed preference graph. It develops a diffusion algorithm similar to Go ogle’s PageRank ( Brin and Page , 1998 ) to deriv e the ranking of no des representing the tennis pla yers. It also pro vides lists for specific playing surfaces and differen t time perio ds. On the basis of the whole dataset, Jimmy Connors w as iden tified as the #1 play er. He is also the winner of the decade 1971-80. F or subsequent y ears, the #1 play ers are Ivan L end l (1981-1990), Pete Sampr as (1991-2000) and R o ger F e der er (2001-2010). The new ranking has a higher predictive p o wer than the official A TP ranking and do es not require arbitrary external criteria, with the exception of a control parameter. Dingle et al. ( 2013 ) use this method to derive PageRank-based tennis rankings instead of the official A TP and WT A rankings. F or top-rank ed play ers, they are broadly similar, but there is a wide v ariation in the tail. The P ageRank-based rankings are found to b e b etter predictor of matc h outcomes. Spanias and Knotten b elt ( 2013 ) presen t tw o new algorithms, SortRank and LadderRank, which make use of a quantitativ e tennis mo del to assess the p erformance of play ers and compare them with each other. Dahl ( 2012 ) introduce a parametric metho d based on linear algebra considering the imp ortance of the matches, too. Motegi and Masuda ( 2012 ) prop ose a net work-based dynamical ranking system, taking into account that the strength of play ers depend on time. The metho d outp erforms b oth the official ranking and Radicchi ( 2011 )’s prestige score in prediction accuracy . Sev eral authors build statistical mo dels with the aim of a go o d prediction p o wer. Clark e and Dyte ( 2000 ) argue that since the rankings are derived from a points rating, an estimate of eac h play er’s chance in a head to head contest can be made from the difference in the play ers’ rating points. Using a year’s tournament results, a logistic regression model can b e fitted to the A TP ratings to estimate that chance. McHale and Morton ( 2011 ) apply a Bradley-T erry t yp e mo del ( Bradley and T erry , 1952 ) to obtain forecasts, and they show that these forecasts 2 are more accurate according to several criteria than the forecasts obtained from standard mo dels emplo yed in the literature. They compare the mo del to tw o logit mo dels, one using official rankings and another using the official ranking p oin ts of the tw o comp eting play ers. Irons et al. ( 2014 ) refine that mo del to b e more transparent, fair and insensitive to bias. As they say , ev en the simplest mo del improv es significantly ov er the current system, despite having three of the same constraints: no surface information is used, only match results count, and a 12 month rolling window is used to w eight games. Ruiz et al. ( 2013 ) apply Data Env elopmen t Analysis. According to their mo del, the ’effi- cien t’ play ers can b e used for the ’inefficien t’ ones as b enc hmark in order to impro ve certain c haracteristics of their pla y . The ranking is based on cross-efficiency ratios. Our paper discusses some theoretical results and applications of the incomplete pairwise com- parison matrices. This section describes the aim of our researc h and reviews sp ort applications with a focus on tennis rankings. The ranking approac h implies the use of pairwise comparisons in a natural wa y . Section 2 provides an ov erview of the results in the area of incomplete pairwise comparison matrices – some of them ha ve been published previously b y the authors of this pap er. The applied mo del for top professional tennis play ers is introduced in the first part of Section 3 together with the description of the database and metho dology . The second part of Section 3 describ es the derived rankings – the Eigenv ector Metho d and the Logarithmic Least Squares Metho d are applied –, and analyzes some prop erties of these results. Section 4 includes further analysis and draws conclusions with some remaining op en questions. 2 Theory and metho ds Our pap er applies the method of pairwise comparisons. Definition 2.1. Pairwise comparison matrix : L et R n × n + denote the class of n × n matric es with p ositive r e al elements. The matrix A =        1 a 12 a 13 . . . a 1 n 1 /a 12 1 a 23 . . . a 2 n 1 /a 13 1 /a 23 1 . . . a 3 n . . . . . . . . . . . . . . . 1 /a 1 n 1 /a 2 n 1 /a 3 n . . . 1        ∈ R n × n + is c al le d a pairwise comparison matrix , if a ii = 1 and a ij = 1 a j i for al l indic es i, j = 1 , . . . , n . In our case the alternatives are tennis play ers. Choosing any t wo of them ( P i and P j ), we ha ve the results of all matches ha ve b een play ed b etw een them. Let the num b er of winning matc hes of P i o ver P j b e x , and the n umber of lost matches y . W e can construct the ratio x i /y i : if it is greater than 1, we can sa y that P i is a ’better’ play er than P j . In case of x i /y i is equal to 1 we are not able to decide who is the b etter. Let the a ij elemen t of the matrix A b e x i /y i , and the a j i elemen t b e y i /x i for all i, j = 1 , . . . , n, i 6 = j . Cho ose the diagonal elements a ii = 1 for all i = 1 , 2 , . . . , n , thus A b ecomes a pairwise comparison matrix ac cording to Definition 2.1 . The PCM matrix A is used to determine a w eight vector w = ( w 1 , w 2 , . . . , w n ) , w i > 0 , ( i = 1 , . . . , n ), where the elements a ij are estimated by w i /w j . Since the estimated v alues are ratios, 3 it is a usual normalization condition that the sum of the weigh ts is equal to 1: P n i =1 w i = 1. That estimation problem can be formulated in several w ays. Saat y ( Saaty , 1980 ) formulated an eigenv alue problem in the Analytic Hierarc hy Pro cess ( AH P ), where the comp onents of the righ t eigenv ector b elonging to the maximal eigenv alue ( λ max ) of matrix A will give the weigh ts. W e will refer to that pro cedure as the Eigenv ector Metho d ( E M ). F or solving the estimation problem it could b e obvious to apply methods based on distance minimization, to o. That approach will estimate the elements of the A matrix with the elemen ts of a matrix W , where the elemen t w ij of W is w i /w j , w i and w j > 0 , ( i, j = 1 , . . . , n ), and the ob jectiv e function to be minimized is the distance of the tw o matrices. Cho o and W edley ( 2004 ) categorized the estimation metho ds and found 12 different distance minimization metho ds of deriving w from A based on minimizing the absolute deviation | a ij − w i /w j | or | w j a ij − w i | , or minimizing the square ( a ij − w i /w j ) 2 or ( w j a ij − w i ) 2 . The effectiveness of some metho ds has b een studied by Lin ( 2007 ). W e will use the Logarithmic Least Squares Metho d ( LLS M ) ( Cra wford and Williams , 1985 ; De Graan , 1980 ; Rabinowitz , 1976 ). Sev eral authors deal with the problem of inconsistency in AHP (see e.g. Bana e Costa and V ansnick ( 2008 )). In our tennis application intransitivit y ma y o ccur, therefore inconsistency is a natural phenomenon. How ev er, the data set is given, consistency correction of the matrix elemen ts could not be done. PCMs may b e inc omplete , that is, they ha ve missing en tries denoted by ∗ : A =        1 a 12 ∗ . . . a 1 n 1 /a 12 1 a 23 . . . ∗ ∗ 1 /a 23 1 . . . a 3 n . . . . . . . . . . . . . . . 1 /a 1 n ∗ 1 /a 3 n . . . 1        . (1) Main results hav e b een discussed by Hark er ( Harker , 1987 ), Carmone, Kara and Zanakis ( Carmone et al. , 1997 ), Kwiesielewicz and v an Uden ( Kwiesielewicz , 1996 ; Kwiesielewicz and v an Uden , 2003 ), Shiraishi, Obata and Daigo ( Shiraishi et al. , 1998 ; Shiraishi and Obata , 2002 ), T akeda and Y u ( T ak eda and Y u , 1995 ), F edrizzi and Giov e ( F edrizzi and Giov e , 2007 ). Definition 2.2. Graph representation of a PCM : Undir e cte d gr aph G := ( V , E ) r epr esents the inc omplete p airwise c omp arison matrix A of size n × n such that V = { 1 , 2 , . . . , n } the vertic es c orr esp ond to the obje cts to c omp ar e and E = { e ( i, j ) | a ij is given and i 6 = j } , that is, the e dges c orr esp ond to the known matrix elements. There are no edges corresponding to the missing elements in the matrix. Kwiesielewicz ( Kwiesielewicz , 1996 ) ha ve considered the Logarithmic Least Squares Metho d ( LLS M ) for incomplete matrices as min X ( i, j ) : a ij is given  log a ij − log  w i w j  2 (2) n X i =1 w i = 1 , (3) w i > 0 , i = 1 , 2 , . . . , n. (4) Theorem 2.1. ( Boz´ oki et al. , 2010 , The or em 4) Optimization pr oblem ( 2 ) - ( 4 ) has a unique solution if and only if G is c onne cte d. F urthermor e, the optimal solution is c alculate d by solving a system of line ar e quations. 4 Note that the incomplete LLS M problem asks for the w eights, ho wev er, missing elemen ts can be calculated as the ratio of the corresponding optimal weigh ts. W e will fo cus only on the w eights. The generalization of the eigen vector metho d to the incomplete case requires t wo steps. First, p ositiv e v ariables x 1 , x 2 , . . . , x d are written instead of missing elemen ts as follo ws: A ( x ) = A ( x 1 , x 2 , . . . , x d ) =        1 a 12 x 1 . . . a 1 n 1 /a 12 1 a 23 . . . x d 1 /x 1 1 /a 23 1 . . . a 3 n . . . . . . . . . . . . . . . 1 /a 1 n 1 /x d 1 /a 3 n . . . 1        , (5) Let x = ( x 1 , x 2 , . . . , x d ) T ∈ R d + . Saat y ( Saaty , 1980 ) defined inconsistency index C R as a p ositive linear transformation of λ max ( A ) suc h that C R ( A ) ≥ 0 and C R ( A ) = 0 if and only if A is consistent. The idea that larger λ max indicates higher C R inconsistency led Shiraishi, Obata and Daigo ( Shiraishi et al. , 1998 ; Shiraishi and Obata , 2002 ) to consider the eigenv alue optimization problem min x > 0 λ max ( A ( x )) . (6) in order to find a completion that minimizes the maximal eigenv alue, or, equiv alently , C R . As in case of incomplete LLS M , uniqueness is closely related to the connectedness of G . Theorem 2.2. ( Boz´ oki et al. , 2010 , The or em 2, Cor ol lary 2 and Se ction 5) Optimization pr oblem ( 6 ) has a unique solution if and only if G is c onne cte d. F urthermor e, ( 6 ) c an b e tr ansforme d to a c onvex optimization pr oblem that c an b e solve d efficiently. Second step is to apply the eigen vector method to the completed pairwise comparison matrix. P arallel with publishing the first theoretical results on incomplete P C M s our research team ha ve b een seeking for applications. The world of sp orts pro vided us a prosp erous exp erimental field. Cs at´ o ( 2013 ) has analysed the chess olympiad. A research pap er was published later as a chapter in a b o ok in Hungarian ( T emesi et al. , 2012 ) on ranking tennis play ers. Some early results hav e b een published and some research questions hav e b een formulated there. The recen t article expands the scop e of the researc h and reports new results. 3 Calculation and results Our aim is to demonstrate that it is possible to compare play ers from a long p erio d of time. There are several options how to choose from the list of professional play ers included in the A TP database. All choices hav e pros and cons. There is no ideal set of play ers and comparative time p erio ds, b ecause exp erts hav e disputes on contro versial issues: Who can represen t a certain era? Ma y w e compare re sults from different perio ds of the carrier path of an individual play er? May w e set up a unified ranking or it is b etter to hav e separate rankings for different surfaces? W e ha ve chosen those 25 pl ay ers who hav e been #1 on the A TP ranking for an y perio d of time from 1973. Figure 1 shows them together with their active p erio d in the world of professional tennis. In our calculations the initial data are as follows: • z ij ( i, j = 1 , . . . , n, i 6 = j ): the num b er of matches ha ve b een play ed b etw een play ers P i and P j ( z ij = z j i ); 5 Figure 1: Length of professional tennis career for the chosen pla yers 1965 1970 1975 1980 1985 1990 1995 2000 2005 2010 Djoko vic Nadal Roddick F ederer F errero Hewitt Safin Kuerten Moy a Rios Kafelniko v Rafter Courier Sampras Agassi Muster Beck er Edberg Wilander Lendl McEnroe Borg Connors Nastase Newcom b e Djoko vic Nadal Roddick F ederer F errero Hewitt Safin Kuerten Moy a Rios Kafelniko v Rafter Courier Sampras Agassi Muster Beck er Edberg Wilander Lendl McEnroe Borg Connors Nastase Newcom b e Y ears • x ij ( i > j ): the num b er of matc hes b etw een play ers P i and P j , where P i w as the winner; • y ij = z ij − x ij ( i > j ): the num b er of matches betw een play ers P i and P j , where P i lost against P j . Definition 3.1. Pairwise c omp arison matrix of top tennis players : p ij elements of matrix P ar e c alculate d fr om the initial data as • p ij = x ij /y ij if i, j = 1 , . . . , n , i > j and x ij 6 = 0 , y ij 6 = 0 ; • p j i = y ij /x ij = 1 /p ij if i, j = 1 , . . . , n , i < j and x ij 6 = 0 , z ij 6 = 0 ; • p ii = 1 for al l i = 1 , . . . , n ; • p ij and p j i elements ar e missing otherwise. A consequence of the definition is that in case of z ij = 0 for at least one pair of the pla yers, the pairwise comparison matrix is incomplete. The in terpretation of p ij > 0 is that the i th pla yer is p ij times b etter than the j th play er. W e ha ve to note that Definition 3.1 is strict in the sense that p ij is also missing in the case when z ij 6 = 0, but one of its comp onent is 0 (either x ij = 0 or y ij = 0). Ho wev er, it can happ en 6 that P i w on sev eral times ov er P j , and he has nev er b een defeated. According to the definition w e can eliminate all pairs where that phenomenon o ccurs, but it w ould b e unfair for the winner pla yer in the giv en pair. Therefore we decided to use artificial p ij v alues for these cases. F or instance p ij = 5 was used if z ij w as less than 5 and y ij = 0, p ij = 10 if the num b er of matc hes w as b etw een 6 and 10, and y ij = 0, and so on. In our calculations we will use this correction metho d and we will refer to it with a subscript 1. Another correction method for p ij could b e that the v alue of p ij = x ij + 2 if y ij = 0. In our calculations w e will refer to that correction metho d with a subscript 2 (see T able 2 and 3 later). One can naturally argue that the choice of p ij is crucial to get different results. W e ha ve made a series of calculations for v arious n umbers of pla yers with sev eral correction v alues ( T emesi et al. , 2012 ) and w e hav e found that the results did not alter significantly . T able 1 c on tains the results of the matches play ed betw een pla yers P i and P j (sum of the symmetric elements of the matrix is z ij for all play ers). W e can see that there are cases when t wo other play ers ha ve play ed more than 30 times with eac h other, and it w as also p ossible that t wo pla yers met less than 5 times. There is a need for balancing the impact of extremely differing matc h num b ers resulted in a wide range of ratios. In order to handle that problem w e introduced a transformation for the elemen ts of p ij : t ij = p z ij / max z ij ij (7) where the transforming factor is the ratio of the num b er of matches b etw een eac h other divided b y the maximum n umber of matches of all pairs. Note that if all play ers hav e the same num b er of matches, transformation ( 7 ) results in t ij = p ij . It approximates 1 when the tw o play ers hav e play ed a small n umber of matches against each other, therefore the outcome seems to b e ’unreliable’. F or instance, the original p ij v alue for the pair A gassi - Be cker w as 10 / 4 = 2 . 5, the transformed t ij v alue is (10 / 4) 14 / 39 = 1 . 3895 where 14 is the n umber of matches betw een A gassi and Be cker , and 39 is the maximum of z ij v alues ( Djokovic vs. Nadal ). The vertices of the graph in Figure 2 represent the play ers. The edges show that the tw o pla yers play ed at least one match against each other. The b old lines connected to the no de lab elled A gassi illustrate that he play ed against 20 of our play ers during his carrier: the degree of the v ertex is 20 (whic h is also the maximum degree). Edges from A gassi to other play ers (e.g. to Connors ) mean that A gassi has more wins than losses against them (indicated also in the neigh b ouring table). Similarly , edges to A gassi from other play ers (e.g. from Rios ) mean that A gassi has more losses than wins against them, while dashed lines represen t an equal n umber of wins and losses (e.g. to Safin ). W e hav e plotted the graphs b elonging to all play ers in an Online App endix, av ailable at http://www.sztaki.mta.hu/ ~ bozoki/tennis/appendix.pdf . Ha ving the incomplete pairwise comparison matrices for the 25 top tennis play ers from T able 1 w e can calculate the weigh t v ectors if the corresponding matrix T is connected. It can be c heck ed that this condition is met: the 20 edges adjacent to the no de A gassi together with the edges Nadal - Djokovic , Newc omb e - Nastase , Nastase - Connors and Connors - Bor g form a spanning tree (see Figure 2 ). 7 T able 1: Pairwise comparisons: n umber of wins/total n umber of matches Agassi Beck er Borg Connors Courier Djoko vic Edberg F ederer F errero Hewitt Kafelniko v Kuerten Lendl McEnroe Moy a Muster Nadal Nastase Newcom b e Rafter Rios Roddick Safin Sampras Wilander T otal Agassi 10/14 2/2 5/12 6/9 3/11 2/5 4/8 8/12 7/11 2/8 2/4 3/4 5/9 0/2 10/15 1/3 5/6 3/6 14/34 5/7 97/182 Beck er 4/14 6/6 6/7 25/35 1/1 4/6 10/21 8/10 2/4 2/3 1/1 2/3 3/5 0/1 7/19 7/10 88/146 Borg 15/23 6/8 7/14 10/15 1/4 1/1 40/65 Connors 0/2 0/6 8/23 0/3 6/12 13/34 14/34 12/27 2/4 0/2 0/5 55/152 Courier 7/12 1/7 3/3 6/10 1/6 1/1 0/4 2/3 2/3 7/12 0/1 0/3 0/3 1/2 4/20 35/90 Djoko vic 15/31 2/3 6/7 2/4 17/39 4/9 0/2 46/95 Edberg 3/9 10/35 6/12 4/10 1/3 14/27 6/13 1/1 10/10 3/3 1/1 6/14 9/20 74/158 F ederer 8/11 16/31 10/13 18/26 2/6 1/3 7/7 10/32 0/3 2/2 21/24 10/12 1/1 106/171 F errero 3/5 1/3 3/13 4/10 1/3 3/5 8/14 2/9 2/3 3/4 0/5 6/12 36/86 Hewitt 4/8 0/1 1/7 8/26 6/10 7/8 3/4 7/12 4/10 3/4 3/5 7/14 7/14 5/9 65/132 Kafelniko v 4/12 2/6 5/6 2/3 4/6 2/3 1/8 5/12 3/6 1/5 3/5 6/8 2/4 2/13 1/2 43/99 Kuerten 4/11 0/1 2/3 2/5 1/4 7/12 4/7 3/3 4/8 2/4 1/2 4/7 1/3 35/70 Lendl 6/8 11/21 2/8 21/34 4/4 13/27 21/36 4/5 1/1 0/1 3/8 15/22 101/175 McEnroe 2/4 2/10 7/14 20/34 1/3 7/13 15/36 6/9 1/2 0/3 7/13 68/141 Moy a 1/4 2/4 1/3 2/4 0/1 0/7 6/14 5/12 3/6 3/7 4/8 2/8 3/4 2/7 1/5 4/7 1/4 40/105 Muster 4/9 1/3 5/12 0/10 4/5 0/3 1/5 4/8 0/3 3/4 0/1 2/11 0/2 24/76 Nadal 2/2 22/39 22/32 7/9 6/10 6/8 7/10 2/2 74/112 Nastase 0/1 5/15 15/27 1/1 0/1 3/9 4/5 0/1 28/60 Newcom b e 3/4 2/4 1/2 1/5 7/15 Rafter 5/15 1/3 3/3 0/3 3/3 1/3 1/4 2/5 4/8 1/1 1/4 3/3 2/3 1/1 4/16 1/3 33/78 Rios 2/3 2/5 3/3 0/1 0/2 1/4 2/5 2/8 2/4 5/7 1/4 1/3 0/2 1/4 0/2 22/57 Roddick 1/6 5/9 3/24 5/5 7/14 1/2 4/5 3/10 2/2 4/7 2/3 37/87 Safin 3/6 1/1 1/2 2/2 2/12 6/12 7/14 2/4 3/7 3/7 1/1 0/2 0/1 3/4 3/7 4/7 41/89 Sampras 20/34 12/19 2/2 16/20 8/14 0/1 4/9 11/13 2/3 5/8 3/3 3/4 9/11 12/16 2/2 1/3 3/7 2/3 115/172 Wilander 2/7 3/10 0/1 5/5 11/20 1/2 7/22 6/13 2/2 1/1 2/3 1/3 41/89 8 Figure 2: Graph represen tation of matrix T New combe Nastase Connors Borg McEnro e Lendl Wilander Edb erg Bec ker Muster Agassi Courier Sampras Rafter Kafelnik ov Rios Mo ya Kuerten Safin F errero Hewitt F ederer Ro ddic k Nadal Djok ovic Agassi against Win Loss Bec ker 10 4 Connors 2 0 Courier 5 7 Edb erg 6 3 F ederer 3 8 F errero 2 3 Hewitt 4 4 Kafelnik ov 8 4 Kuerten 7 4 Lendl 2 6 McEnro e 2 2 Mo ya 3 1 Muster 5 4 Nadal 0 2 Rafter 10 5 Rios 1 2 Ro ddic k 5 1 Safin 3 3 Sampras 14 20 Wilander 5 2 Sum 97 85 T able 2: Rankings E M 2 LLS M 2 E M W 2 LLS M W 2 W / L Nadal 1 1 1 1 2 F ederer 2 2 2 2 3 Sampras 3 3 3 3 1 Lendl 11 8 4 4 6 Borg 13 11 6 5 4 Bec ker 4 4 5 6 5 Djok ovic 5 5 7 7 10 Agassi 9 9 8 8 7 Hewitt 6 7 9 9 9 Kuerten 16 15 10 10 8 Safin 12 10 11 11 15 McEnro e 20 18 12 12 11 Nastase 22 20 14 13 13 F errero 17 16 16 14 20 Ro ddic k 8 6 13 15 18 Wilander 15 14 17 16 16 Rios 21 22 18 17 22 Rafter 7 13 15 18 19 New combe 23 21 21 19 14 Kafelnik ov 14 17 19 20 17 Mo ya 19 19 22 21 23 Edb erg 10 12 20 22 12 Courier 18 23 23 23 21 Muster 24 24 24 24 25 Connors 25 25 25 25 24 W eight vectors hav e b een computed with the Logarithmic Least Squares Metho d ( LLS M in T able 2 ) and with the Eigenv ector Metho d ( E M ) as it was describ ed in Section 2 . On the basis of the weigh t vectors, eigh t rankings hav e b een calculated without and with transformation (in the latter case w e used the subscript W for iden tification), and different correction metho ds ha ve also b een applied (subscripts 1 and 2 as it w as introduced earlier). Selected results are demonstrated in T able 2 . The fourth column, LLS M W 2 , for example, is a ranking given b y Logarithmic Least Squares Metho d with the second correction pro cedure and transformed data. Note that the play ers are listed in T able 2 according to this ranking. The fifth column includes the ranking according to the win to loss ratio, indicated by W / L. Rankings were practically the same with both estimation metho ds, as it can be seen from T able 2 . The impact of the correction method is not significan t either. (That w as the reason wh y T able 2 do es not contains calculations with the first t yp e correction.) The v alues of the Sp earman rank correlation co efficients in T able 3 supp ort these prop ositions: the elements of the top-left and b ottom-right 4 × 4 submatrices are close to the iden tity matrix. The correlation co efficients – comparing rankings with the same estimation metho d – suggest that filtering the impact of differences in the total matc h n umbers eliminated the minor impact of the correction methods, to o. Analysing the impact of the estimation metho ds and v arious forms of data correction the authors had similar exp erience with 34 top pla yers ( T emesi et al. , 2012 ). Ho wev er, data transformation ( 7 ) may change the rankings significantly , as it can b e seen in 10 T able 3: Sp earman rank correlation coefficients E M 1 E M 2 LLS M 1 LLS M 2 E M W 1 E M W 2 LLS M W 1 LLS M W 2 E M 1 1 0.9715 0.9269 0.9154 0.7546 0.7423 0.6869 0.6631 E M 2 0.9715 1 0.9677 0.9569 0.8015 0.7908 0.7385 0.7177 LLS M 1 0.9269 0.9677 1 0.9915 0.8638 0.8469 0.8085 0.7946 LLS M 2 0.9154 0.9569 0.9915 1 0.8931 0.8831 0.8446 0.8338 E M W 1 0.7546 0.8015 0.8638 0.8931 1 0.9962 0.9908 0.9854 E M W 2 0.7423 0.7908 0.8469 0.8831 0.9962 1 0.9900 0.9877 LLS M W 1 0.6869 0.7385 0.8085 0.8446 0.9908 0.9900 1 0.9969 LLS M W 2 0.6631 0.7177 0.7946 0.8338 0.9854 0.9877 0.9969 1 T able 3 , to o. The corresp onding rank c orrelation co efficients in the top-righ t and bottom-left 4 × 4 submatrices confirm this statement. According to our in terpretation the v alue judgemen t of the ranking exp ert determines the choice b etw een these rankings. Therefore if the exp ert’s opinion is that a ratio of 2 has to b e represented in differen t wa ys if it was resulted from 6 matc hes (4 : 2) or from 30 matches (20 : 10) than the recommended normalization has to b e implemen ted and the corresponding ranking can be c hosen. T urning back to T able 2 , the first three play ers ( Nadal , F e der er , Sampr as ) and the last three pla yers ( Courier , Muster , Connors ) are the same in b oth rankings. Some differences in the rank n umbers can b e found in other parts of the list. T ennis fans can debate the final ranking, of course. One can compare these rankings to the win /loss ratio of the play ers, given in the ninth column of T able 2 . How ever, the most important fact is that the T op 12 includes big names from the recent championships and from the go o d old times, as well. The conclusion is that it is p ossible to pro duce rankings based on pairwise comparisons and ov erarching four decades with pla yers who hav e nev er met on the court. 4 Conclusions and op en questions In case of ha ving historical data incomplete pairwise comparison matrices can be applied in order to answer the question: what is the ranking of the play ers for a long time p erio d? Who is the #1 play er? With this metho dology it is p ossible to use face to face match results. V arious t yp es of transformations can mo dify the original data set with the in tention of correcting either data problems or biasing factors. How ever, we did not take into account the impact of the carrier path of a pla yer. Ev ery match had identical weigh t without considering its p osition on the time line. Different surfaces did not pla y specific role, either. Ha ving had a great num b er of calculations with tennis results w e hav e b een in terested in finding answer for the question ’What are those prop erties of matrix T which hav e an impact on the ranking?’ Ranking can dep end on the num b er and the distribution of the comparisons. The num b er of comparisons can b e characterized b y the density (sparsit y) of the PCM. F or a fully completed PCM the density of the matrix is 1. Low er v alues mean that the matrix is incomplete. In our case the density of T is 341 / 625 ≈ 0 . 5456 since 341 elements are known in the 25 × 25 matrix. Another indicator of the structure of matrix T is the distribution of elemen ts, whic h can b e c haracterized by the degree of v ertices in the graph representation of the PCM. W e ha ve designed a tool for exploring the connectedness of the incomplete pairwise comparison matrices visually . Figure 3 sho ws the distribution of degrees in our case, which can b e chec ked in the Online App endix ( http://www.sztaki.mta.hu/ ~ bozoki/tennis/appendix.pdf ), where clic king on a 11 no de shows the edges adjacen t to it. The maximum degree is 20 in the case of A gassi . Figure 3: Degree of vertices in the graph representation of matrix T Agassi Sampras Mo ya Bec ker Rafter Safin Courier Kafelnik ov Rios Hewitt Edb erg F ederer Kuerten Muster F errero Lendl Wilander Connors McEnro e Ro ddic k Nadal Nastase Djok ovic Borg New combe 0 2 4 6 8 10 12 14 16 18 20 22 Degree Moreo ver, increasing the num b er of matches pla yed b et ween those who hav e b een play ed with each other leav es the v alue of density and the degree of v ertices unc hanged but the ranking can change as a result of the estimation method. An interesting question is ’How an additional matc h with a given result affects the ranking?’ The impact of low er and higher v alues of sparsity (degrees of v ertices) can also b e analysed. A challenging question could be ’Whic h play er can b e cancelled without changing the ranking?’ The inconsistency of pairwise comparison matrices plays an important role b oth in theory and practice ( K ´ eri , 2011 ). F urther researc h includes the analysis of inconsistency of incomplete pairwise comparison matrices of large size. In our case we cannot sp eak of the inconsistency of a decision mak er since the matrix elemen ts originate from tennis matches, w e might also say: from life. Intransitiv e triads ( A b eats B , B b eats C , and C beats A ) o ccur often in sp orts. W e hav e found 50 intransitiv e triads in our example, they are plotted in the Online Appendix, a v ailable at http://www.sztaki.mta.hu/ ~ bozoki/tennis/appendix.pdf . W e hop e to return to the problem of analysing intransitiv e triads in a(n incomplete) pairwise comparison matrix, or, equiv alen tly , in the directed graph asso ciated. 12 Ac kno wledgemen t The authors are grateful to the anonymous review ers for v aluable and constructive recommen- dations. Research was supp orted in part by OTKA grants K 77420 and K 111797. References Bana e Costa, C. A. and V ansnick, J.-C. (2008). A critical analysis of the eigen v alue metho d used to derive priorities in AHP. Eur op e an Journal of Op er ational R ese ar ch , 187(3):1422–1428. Boz´ oki, S., F¨ ul¨ op, J., and R´ on yai, L. (2010). On optimal completion of incomplete pairwise comparison matrices. Mathematic al and Computer Mo del ling , 52(1-2):318–333. Bradley , R. A. and T erry , M. E. (1952). Rank analysis of incomplete blo ck designs: I. The metho d of paired comparisons. Biometrika , 39(3/4):324–345. Brin, S. and Page, L. (1998). The anatomy of a large-scale hypertextual web searc h engine. Computer networks and ISDN systems , 30(1):107–117. Carmone, F., Kara, A., and Zanakis, S. H. (1997). A Mon te Carlo inv estigation of incomplete pairwise comparison matrices in AHP. Eur op e an Journal of Op er ational R ese ar ch , 102(3):538– 553. Cho o, E. U. and W edley , W. C. (2004). A common framework for deriving preference v alues from pairwise comparison matrices. Computers & Op er ations R ese ar ch , 31(6):893–908. Clark e, S. R. and Dyte, D. (2000). Using official ratings to simulate ma jor tennis tournamen ts. International T r ansactions in Op er ational R ese ar ch , 7(6):585–594. Cra wford, G. and Williams, C. (1985). A note on the analysis of sub jective judgment matrices. Journal of Mathematic al Psycholo gy , 29(4):387–405. Csat´ o, L. (2013). Ranking by pairwise comparisons for Swiss-system tournaments. Centr al Eur op e an Journal of Op er ations R ese ar ch , 21(4):783–803. Dahl, G. (2012). A matrix-based ranking metho d with application to tennis. Line ar A lgebr a and its Applic ations , 437(1):26–36. De Graan, J. G. (1980). Extensions of the m ultiple criteria analysis metho d of T. L. Saaty . V o orburg. National Institute for W ater Supply . Dingle, N., Knottenbelt, W., and Spanias, D. (2013). On the (Page) Ranking of professional tennis play ers. In T ribastone, M. and Gilmore, S., editors, Computer Performanc e Engine ering , Lecture Notes in Computer Science, pages 237–247. Springer Berlin Heidelberg. F edrizzi, M. and Gio ve, S. (2007). Incomplete pairwise comparison and consistency optimization. Eur op e an Journal of Op er ational R ese ar ch , 183(1):303–313. Hark er, P . T. (1987). Incomplete pairwise comparisons in the analytic hierarch y pro cess. Math- ematic al Mo del ling , 9(11):837–848. Irons, D. J., Buc kley , S., and Paulden, T. (2014). Dev eloping an impro ved tennis ranking system. Journal of Quantitative Analysis in Sp orts , 10(2):109–118. 13 K ´ eri, G. (2011). On qualitatively consisten t, transitive and contradictory judgmen t matrices emerging from multiattribute decision pro cedures. Centr al Eur op e an Journal of Op er ations R ese ar ch , 19(2):215–224. Kwiesielewicz, M. (1996). The logarithmic least squares and the generalized pseudoin verse in estimating ratios. Eur op e an Journal of Op er ational R ese ar ch , 93(3):611–619. Kwiesielewicz, M. and v an Uden, E. (2003). Ranking decision v ariants by sub jectiv e paired comparisons in cases with incomplete data. In Kumar, V., Ga vrilov a, M. L., T an, C. J. K., and L’Ecuyer, P ., editors, Computational Scienc e and Its Applic ations – ICCSA 2003 , volu me 2669 of L e ctur e Notes in Computer Scienc e , pages 208–215. Springer, Berlin Heidelberg. Lin, Y.-K. (2007). On a multicommodity sto c hastic-flow netw ork with unreliable no des sub ject to budget constraint. Eur op e an Journal of Op er ational R ese ar ch , 176(1):347–360. McHale, I. and Morton, A. (2011). A Bradley-Terry t yp e mo del for forecasting tennis match results. International Journal of F or e c asting , 27(2):619–630. Motegi, S. and Masuda, N. (2012). A net work-based dynamical ranking system for comp etitive sp orts. Scientific r ep orts , 2. Rabino witz, G. (1976). Some commen ts on measuring w orld influence. Conflict Management and Pe ac e Scienc e , 2(1):49–55. Radicc hi, F. (2011). Who is the b est play er ever? A complex netw ork analysis of the history of professional tennis. PloS one , 6(2):e17249. Ruiz, J. L., Pastor, D., and Pastor, J. T. (2013). Assessing professional tennis play ers using Data En velopmen t Analysis (DEA). Journal of Sp orts Ec onomics , 14(3):276–302. Saat y , T. L. (1980). The Analytic Hier ar chy Pr o c ess: planning, priority setting, r esour c e al lo c a- tion . McGraw-Hill, New Y ork. Shiraishi, S. and Obata, T. (2002). On a maximization problem arising from a positive reciprocal matrix in ahp. Bul letin of informatics and cyb ernetics , 34(2):91–96. Shiraishi, S., Obata, T., and Daigo, M. (1998). Properties of a p ositive recipro cal matrix and their application to AHP. Journal of the Op er ations R ese ar ch So ciety of Jap an-Keiei Kagaku , 41(3):404–414. Spanias, D. and Knotten b elt, W. (2013). T ennis play er ranking using quan titative mo dels. Man uscript. http://www.doc.ic.ac.uk/ ~ wjk/publications/spanias- knottenbelt- mis- 2013.pdf . T akeda, E. and Y u, P .-L. (1995). Assessing priorit y weigh ts from subsets of pairwise compar- isons in multiple criteria optimization problems. Eur op e an Journal of Op er ational R ese ar ch , 86(2):315–331. T emesi, J., Csat´ o, L., and Boz´ oki, S. (2012). Mai ´ es r´ egi id˝ ok tenisze – A nem teljesen kit¨ olt¨ ott p´ aros ¨ osszehasonl ´ ıt´ as m´ atrixok egy alk almaz´ asa. In Solymosi, T. and T emesi, J., editors, Egyens´ uly ´ es optimum. T anulm´ anyok F or g´ o F er enc 70. sz¨ ulet´ esnapj´ ar a , pages 213–245. Aula Kiad´ o, Budap est. 14