The work applies the funnel plot methodology to measure and visualize uncertainty in the research performance of Italian universities in the science disciplines. The performance assessment is carried out at both discipline and overall university level. The findings reveal that for most universities the citation-based indicator used gives insufficient statistical evidence to infer that their research productivity is inferior or superior to the average. This general observation is one that we could indeed expect in a higher education system that is essentially non-competitive. The question is whether the introduction of uncertainty in performance reporting, while technically sound, could weaken institutional motivation to work towards continuous improvement.
In the current knowledge-based economy, research and higher education systems play a significant role in supporting the competitiveness and socio-economic growth of nations, through the education of white collar workers and production of new knowledge. Improvement in the research and higher education infrastructure has with good reason become a policy priority for a growing number of governments. Among the interventions to increase the effectiveness and efficiency of research institutions, and the socio-economic returns from public spending on R&D, a growing number of countries are launching national assessment exercises of their research institutions. The exercises are intended to accomplish various aims, as selected by the national governments: for informing selective funding of research institutions; stimulating better research performance; reducing information asymmetry between supply and demand in the market for knowledge; informing policy formulation and strategic decisions; and last but not least, demonstrating that public investment in research is effective and delivers public benefits. An international comparative analysis of performance-based research funding (Hicks, 2012), indicates that subsequent to the example of the original UK research assessment exercise (the RAE, in 1986), at least 14 other countries (China, Australia, New Zealand and 11 in the EU) have chosen to implement national assessment exercises as the basis for directing at least some portion of public financing for research institutions. Alongside this, several annual world university rankings continuously receive great media attention, influencing opinion and practical choices. The various national and international assessments employ a variety of indicators and methods (bibliometric, peer-review, informed peer-review, surveys) to assess institutions' research performance. A common feature of the vast majority of the performance assessments is the lack of confidence intervals indicating the likely range of population values. The scores observed for research performance are in fact related to the assumptions and limits of the particular measurement method and indicators, and in the case of aggregate measures to the different sizes of the research institutions. Accounting for the uncertainty embedded in the measurements is crucial to establish whether the performance of an institution is truly outstanding or the result of random fluctuations.
Assessment of research performance is affected by both bias and uncertainty factors2 . Bias factors generate fluctuations with systematic effects. A typical source of bias is the differing intensity of publication and citation across fields, which the evaluator ideally attempts to limit through a fine-grained classification of scientists and field-normalization of citations. Conversely, uncertainty factors randomly affect the assessment, meaning they will generate fluctuations without systematic effects in favor or against particular groups. Typical factors increasing uncertainty in performance assessment are the variability in intensity of production due to personal events, or due to the patterns characteristic of research projects, or the varying lengths of review and publication time across journals. Ideally, uncertainty factors should again be limited, but they cannot be completely eliminated. Notwithstanding uncertainty, the analyses of research performance can still be valid, as long as the reporting includes measures of uncertainty. The recent Leiden manifesto (Hicks et al., 2015) wisely recommends that practitioners ‘avoid misplaced concreteness and false precision’ in reporting performance values, and that ‘if uncertainty and error can be quantified, for instance using error bars, this information should accompany published indicator values’.
However, indications of uncertainty are generally not provided for the popular international ’league tables’ of universities. This is true whether the performance scores and relevant rankings are produced by ’non-bibliometricians’, such as the Shanghai Jiao Tong University Ranking (SJTU, 2016), QS World University Rankings (QS, 2016) and Times Higher Education World University Rankings (THE, 2016), or whether they are produced by bibliometricians themselves, such as the Scimago Institutions Ranking (Scimago, 2016). The CWTS Leiden Rankings few years ago integrated stability intervals (Waltman et al., 2012). In our studies concerning Italian university research productivity rankings (e.g. Abramo et al., 2011), we have ourselves not usually provided the likely range of performance values. Recently we attempted to deal with this shortcoming, by applying a funnel plot methodology to measure and visualize uncertainty in the research performance of the institutions. The funnel plot shows the uncertainty in data values by adding confidence bands, indicating the range where research performance indicator’s values are expected to lie on the basis of the
This content is AI-processed based on open access ArXiv data.
