A critical comparative analysis of five world university rankings

📝 Abstract

To provide users insight into the value and limits of world university rankings, a comparative analysis is conducted of 5 ranking systems: ARWU, Leiden, THE, QS and U-Multirank. It links these systems with one another at the level of individual institutions, and analyses the overlap in institutional coverage, geographical coverage, how indicators are calculated from raw data, the skewness of indicator distributions, and statistical correlations between indicators. Four secondary analyses are presented investigating national academic systems and selected pairs of indicators. It is argued that current systems are still one-dimensional in the sense that they provide finalized, seemingly unrelated indicator values rather than offering a data set and tools to observe patterns in multi-faceted data. By systematically comparing different systems, more insight is provided into how their institutional coverage, rating methods, the selection of indicators and their normalizations influence the ranking positions of given institutions.

💡 Analysis

To provide users insight into the value and limits of world university rankings, a comparative analysis is conducted of 5 ranking systems: ARWU, Leiden, THE, QS and U-Multirank. It links these systems with one another at the level of individual institutions, and analyses the overlap in institutional coverage, geographical coverage, how indicators are calculated from raw data, the skewness of indicator distributions, and statistical correlations between indicators. Four secondary analyses are presented investigating national academic systems and selected pairs of indicators. It is argued that current systems are still one-dimensional in the sense that they provide finalized, seemingly unrelated indicator values rather than offering a data set and tools to observe patterns in multi-faceted data. By systematically comparing different systems, more insight is provided into how their institutional coverage, rating methods, the selection of indicators and their normalizations influence the ranking positions of given institutions.

📄 Content

1

A critical comparative analysis of five world university rankings Henk F. Moed1 Sapienza University of Rome, Italy Version 15 Nov 2016, accepted for publication in Scientometrics Summary
To provide users insight into the value and limits of world university rankings, a comparative analysis is conducted of 5 ranking systems: ARWU, Leiden, THE, QS and U-Multirank. It links these systems with one another at the level of individual institutions, and analyses the overlap in institutional coverage, geographical coverage, how indicators are calculated from raw data, the skewness of indicator distributions, and statistical correlations between indicators. Four secondary analyses are presented investigating national academic systems and selected pairs of indicators. It is argued that current systems are still one-dimensional in the sense that they provide finalized, seemingly unrelated indicator values rather than offer a dataset and tools to observe patterns in multi-faceted data. By systematically comparing different systems, more insight is provided into how their institutional coverage, rating methods, the selection of indicators and their normalizations influence the ranking positions of given institutions.

1. Introduction
   In most OECD countries, there is an increasing emphasis on the effectiveness and efficiency of government-supported research. Governments need systematic evaluations for optimizing their research allocations, re-orienting their research support, rationalizing research organizations, restructuring research in particular fields, or augmenting research productivity. In view of this, they have stimulated or imposed evaluation activities of their academic institutions. Universities have become more diverse in structure and are more oriented towards economic and industrial needs.

In March 2000, the European Council agreed a new strategic goal to make Europe “the most competitive and dynamic knowledge‐based economy in the world, capable of sustainable economic growth with more and better jobs and greater social cohesion”. Because of the importance of research and development to “generating economic growth, employment and social cohesion”, the Lisbon Strategy says that European universities “must be able to compete with the best in the world through the completion of the European Higher Education Area” (EU Council, 2000). In its resolution ‘Modernizing Universities for Europe‘s Competitiveness in a Global Knowledge Economy’, the European Council expressed the view that the “challenges posed by globalization require that the European Higher Education Area and the European Research Area be fully open to the world and that Europe’s universities aim to become worldwide competitive players” (EU Council, 2007, p. 3).

An Expert Group on the assessment of university-based research noted in 2009 that university rankings have become an increasing influence on the higher education landscape since US News and World Report began providing consumer‐type information about US universities in 1983. They

1 Visiting professor. Email: henk.moed@uniroma1.it; hf.moed@gmail.com

2

“enjoy a high level of acceptance among stakeholders and the wider public because of their simplicity and consumer type information” (AUBR Expert Group, 2009, p 9).

University ranking systems have been intensely debated, for instance by Van Raan (2005), Calero- Medina et al. (2008), Salmi (2009), Hazelkorn (2011), Rauhvargers (2011; n.d.) and Shin, Toutkoushian and Teichler (eds.) (2011). A report from the European University Association concluded that despite their shortcomings, evident biases and flaws, rankings are here to stay. “For this reason it is important that universities are aware of the degree to which they are transparent, from a user’s perspective, of the relationship between what it is stated is being measured and what is in fact being measured, how the scores are calculated and what they mean” (Rauhvargers, 2011, p. 7).

A base notion underlying the current article is that a critical, comparative analysis of a series of university ranking systems can provide useful knowledge that helps a wide range of interested users to better understand the information provided in these systems, and to interpret and use it in an informed, responsible manner. The current article aims to contribute to such an analysis by presenting a study of the following five ranking systems: ARWU World University Rankings 2015, CWTS Leiden Ranking 2016, QS World University Rankings 2015-2016, THE World University Rankings 2015-2016, and U-Multirank 2016 Edition. An overview of the indicators included in the various systems is given in Table A1 in the Appendix.

ARWU, the Academic Ranking of World Universities, also indicated as ‘Shanghai Ranking’ is the oldest ranking system. Initially created by the Center for World-Class Univ

This content is AI-processed based on ArXiv data.