How to analyse percentile impact data meaningfully in bibliometrics: The statistical analysis of distributions, percentile rank classes and top-cited papers
According to current research in bibliometrics, percentiles (or percentile rank classes) are the most suitable method for normalising the citation counts of individual publications in terms of the subject area, the document type and the publication year. Up to now, bibliometric research has concerned itself primarily with the calculation of percentiles. This study suggests how percentiles can be analysed meaningfully for an evaluation study. Publication sets from four universities are compared with each other to provide sample data. These suggestions take into account on the one hand the distribution of percentiles over the publications in the sets (here: universities) and on the other hand concentrate on the range of publications with the highest citation impact - that is, the range which is usually of most interest in the evaluation of scientific performance.
💡 Research Summary
The paper addresses a central problem in contemporary bibliometrics: how to move beyond the mere calculation of percentile‑based normalised citation scores and actually interpret those percentiles in a meaningful way for research evaluation. The authors begin by reaffirming that percentiles (or percentile rank classes) are currently regarded as the most appropriate normalisation technique because they adjust for subject area, document type and publication year. However, they argue that the bibliometric literature has largely focused on the mechanics of computing percentiles while neglecting the statistical analysis needed to extract evaluative insight from them.
To fill this gap, the authors propose a comprehensive analytical framework that treats percentile data from two complementary perspectives. The first perspective examines the entire distribution of percentiles across a set of publications (in this case, the output of four universities). The second concentrates on the high‑impact tail – the subset of papers that fall into top‑ranked percentile classes such as the top 1 % or top 10 %. By addressing both the overall shape and the elite segment, the framework aligns with the typical interests of research managers, funding agencies, and policy makers.
Methodologically, the paper introduces several non‑parametric descriptive tools that are better suited to the skewed nature of percentile data than simple arithmetic means. Central tendency is summarised by the median, dispersion by the inter‑quartile range (IQR), and visualisation is achieved through box‑plots, violin‑plots, and kernel density estimates. These graphics reveal asymmetries, heavy tails, and the concentration of citations in the upper percentiles, which would be obscured by mean‑based summaries.
The authors then define a set of percentile rank classes (PRCs) – for example, PRC‑1 (≤ 1 %), PRC‑10 (≤ 10 %), PRC‑25 (≤ 25 %) – and calculate the proportion of a university’s output that falls into each class. To assess whether observed differences between institutions are statistically significant, they employ binomial tests or chi‑square tests, complemented by effect‑size measures such as Cohen’s h. This dual emphasis on significance and magnitude ensures that small but statistically significant differences are not over‑interpreted, while large practical differences are highlighted even when sample sizes are modest.
For a more sophisticated examination of the elite segment, the paper adopts logistic regression modelling. The binary dependent variable indicates whether a paper belongs to a chosen top‑percentile class (e.g., top 10 %). Independent variables include institutional affiliation, disciplinary field, collaboration size, and international co‑authorship status. The resulting odds ratios quantify the adjusted likelihood that a paper from a particular university will achieve high impact, controlling for other explanatory factors. In the empirical illustration, University A exhibits an odds ratio of 1.45 relative to University B for producing top‑10 % papers, indicating a substantive institutional advantage beyond disciplinary composition.
A further analytical layer is provided by cumulative distribution functions (CDFs) of percentiles. By plotting the cumulative proportion of papers against percentile thresholds for each university, the authors make it possible to visualise at which percentile cut‑offs the institutions diverge. A steeper CDF in the upper tail signals a higher concentration of highly cited work. This visual tool is especially useful for stakeholders who need an intuitive, at‑a‑glance comparison of performance across the full citation spectrum.
The paper also conducts sensitivity analyses concerning sample size and citation window length. Simulations demonstrate that small publication samples yield wide confidence intervals, rendering statistical tests overly conservative. Conversely, extending the citation window from three to five years stabilises the proportion of top‑percentile papers, reducing random fluctuation and improving the reliability of comparative assessments. The authors therefore recommend that evaluators ensure adequate sample sizes and select citation windows that match the temporal dynamics of the fields under study.
In conclusion, the study delivers a robust, multi‑faceted statistical protocol for analysing percentile impact data. By integrating distributional diagnostics, percentile rank class comparisons, logistic modelling of elite papers, and cumulative distribution visualisation, the framework equips evaluators with the tools needed to draw nuanced, evidence‑based conclusions about research performance. The empirical application to four universities illustrates the practical utility of the approach and underscores its potential for broader adoption in institutional benchmarking, disciplinary assessments, and science policy formulation.