Predicting results of the Research Excellence Framework using departmental $h$-Index

We compare estimates for past institutional research performances coming from two bibliometric indicators to the results of the UK’s Research Assessment Exercise which last took place in 2008. We demonstrate that a version of the departmental h-index is better correlated with the actual results of that peer-review exercise than a competing metric known as the normalised citation-based indicator. We then determine the corresponding h-indices for 2008-2013, the period examined in the UK’s Research Excellence Framework (REF) 2014. We place herewith the resulting predictions on the arXiv in advance of the REF results being published (December 2014). These may be considered as unbiased predictions of relative performances in that exercise. We will revisit this paper after the REF results are available and comment on the reliability or otherwise of these bibliometrics as compared with peer review.

💡 Research Summary

The paper investigates whether bibliometric indicators can serve as reliable proxies for peer‑review based research assessments in the United Kingdom, focusing on the 2008 Research Assessment Exercise (RAE) and the forthcoming 2014 Research Excellence Framework (REF). Two metrics are examined: a departmental version of the Hirsch index (h‑index) and the Normalised Citation Impact (NCI) supplied by Thomson Reuters Research Analytics. Using Scopus data, the authors compile publication records for UK higher‑education institutions (HEIs) in four disciplines—biology, chemistry, physics, and sociology—restricted to papers authored by staff affiliated with each institution during the relevant assessment windows (2001‑2007 for RAE, 2008‑2013 for REF).

The departmental h‑index is defined as the largest number h such that the department has produced h papers each cited at least h times within the assessment period. The authors calculate an “immediate” h‑index (h2008) based on publications up to the end of 2007, as well as lagged versions (h2009, h2010, etc.) to explore the effect of citation accrual over time. The NCI is a field‑normalised citation metric that compares a department’s average citation rate to the expected rate for its discipline, thereby attempting to control for differences in citation practices across fields.

To assess predictive power, the study correlates each bibliometric indicator with three peer‑review derived scores: the weighted quality score s (derived from the official RAE formula), an alternative weighted score s′ (reflecting a post‑political lobbying adjustment), and s_output (which excludes environment and esteem components). Pearson correlation coefficients and Spearman rank correlations are computed for each pairing.

Results show that the departmental h‑index consistently outperforms NCI. The highest Pearson correlation (≈0.78) occurs between h2008 and s, indicating a strong linear relationship between the immediate h‑index and the official quality measure. NCI correlations never exceed 0.5, suggesting that even after normalisation it captures less of the variance explained by peer review. Visualisations confirm that h‑index rankings align closely with s and s_output rankings across all four disciplines.

Based on these findings, the authors calculate departmental h‑indices for the 2008‑2013 period and generate a set of predicted REF 2014 rankings, which they posted on arXiv prior to the official REF results release in December 2014. They propose to revisit the paper after the REF outcomes become available, comparing the predictions to the actual rankings to evaluate the robustness of the h‑index as a forecasting tool.

The paper acknowledges several limitations. Scopus affiliation data sometimes omit certain UK institutions, leading to incomplete h‑index coverage for a few universities (e.g., Open University). Moreover, the set of papers used for bibliometric calculations does not perfectly match the subset submitted for RAE/REF, potentially diluting the observed correlations. Finally, the h‑index’s cumulative nature introduces a citation lag, meaning that recent high‑impact work may not be fully reflected until later years.

In conclusion, the study provides empirical evidence that a departmental h‑index is a more reliable indicator of research quality, as judged by peer review, than a normalised citation impact metric. The authors argue that the h‑index could be employed as a supplementary tool in research assessment exercises, offering a cost‑effective, data‑driven complement to costly peer‑review panels. Future work will involve a post‑hoc validation against REF 2014 results and exploration of more sophisticated, possibly multivariate, bibliometric models to further enhance predictive accuracy.