Universality of Performance Indicators based on Citation and Reference Counts

We find evidence for the universality of two relative bibliometric indicators of the quality of individual scientific publications taken from different data sets. One of these is a new index that considers both citation and reference counts. We demonstrate this universality for relatively well cited publications from a single institute, grouped by year of publication and by faculty or by department. We show similar behaviour in publications submitted to the arXiv e-print archive, grouped by year of submission and by sub-archive. We also find that for reasonably well cited papers this distribution is well fitted by a lognormal with a variance of around 1.3 which is consistent with the results of Radicchi, Fortunato, and Castellano (2008). Our work demonstrates that comparisons can be made between publications from different disciplines and publication dates, regardless of their citation count and without expensive access to the whole world-wide citation graph. Further, it shows that averages of the logarithm of such relative bibliometric indices deal with the issue of long tails and avoid the need for statistics based on lengthy ranking procedures.

💡 Research Summary

The paper investigates whether two relative bibliometric indicators—one based solely on citation counts (c_f) and a newly proposed one that combines citations with reference counts (c_r)—exhibit a universal statistical behavior across disparate scientific fields and publication years. Building on the earlier work of Radicchi, Fortunato, and Castellano (2008), which showed that the normalized citation count c_f = c/c₀ follows a log‑normal distribution with a variance around 1.3 regardless of discipline, the authors aim to confirm this universality using much smaller, more readily available data sets and to extend it to a metric that does not require global citation databases.

The first indicator, c_f, is defined as the ratio of a paper’s citation count c to the average citation count c₀ of all papers published in the same year and field. The second indicator, c_r, is defined as (c/r) divided by the average of c/r over a chosen set S, where r denotes the number of references in the paper. By normalizing citations with reference counts, c_r penalizes review articles and other works that accrue many citations simply because they contain extensive bibliographies, thereby offering a more balanced measure of impact. Importantly, c_r can be computed from the citation and reference data that are typically available in commercial databases (e.g., Web of Science) without needing the worldwide citation graph required for c₀.

Two empirical data sets are examined. The first consists of all publications from 1997–2007 authored by at least one permanent staff member of Imperial College London that have both a positive citation count and a positive reference count. After filtering out papers with zero citations or references, roughly 10,800 papers remain. These papers are grouped in two ways: (i) by academic faculty (Natural Sciences, Medicine, Engineering) and publication year, and (ii) by department over three‑year windows. The second data set comprises pre‑prints submitted to the arXiv repository, grouped by sub‑archive (e.g., astro‑ph, hep‑th) and year of submission.

For each group S, the authors compute c_f and c_r for every paper, take natural logarithms, and fit the resulting distributions to a log‑normal probability density function F(x; μ, σ²) with the constraint μ = −σ²/2 (so that the mean of the original variable equals one). Maximum‑likelihood estimation yields σ² values consistently between 1.2 and 1.5 across faculties, years, and arXiv sub‑archives, closely matching the σ² ≈ 1.3 reported by Radicchi et al. Goodness‑of‑fit is assessed with χ² tests; most fits produce χ² per degree of freedom well below critical values, indicating that a single‑parameter log‑normal model adequately captures the bulk of the data.

The analysis also reveals systematic deviations at the distribution tails. Papers with very low citation counts (roughly the bottom 10 % of the mean for a given faculty) fall below the log‑normal prediction, suggesting that the mechanisms driving occasional citations to obscure or niche works differ from those governing highly cited papers. Conversely, the extreme right tail (highly cited papers) shows larger fluctuations, likely due to the small number of such items. The authors note that review articles, which typically have many references, receive lower c_r scores, confirming that the reference‑based normalization effectively mitigates over‑valuation of such papers.

A key practical contribution is the demonstration that averaging the logarithms of c_f or c_r (i.e., computing ⟨log c_f⟩ or ⟨log c_r⟩) provides a robust, scale‑independent summary statistic. Because the log‑normal distribution has a finite mean and variance, these averages are not dominated by the heavy tails that plague raw citation counts. Consequently, comparisons between institutions, departments, or time periods can be made without resorting to lengthy ranking procedures or accessing the full worldwide citation network.

The paper acknowledges limitations. Zero‑cited papers cannot be included in the log‑transformation and are excluded from the analysis, which may bias results for fields with many uncited works. The study also relies on the accuracy of reference counts in the underlying database; books and non‑journal sources are counted as references but do not contribute citations, potentially introducing minor inconsistencies.

In conclusion, the authors provide strong empirical evidence that both the traditional citation‑only indicator and the novel citation‑to‑reference indicator follow a universal log‑normal distribution with a variance near 1.3 across diverse scientific domains and publication years. The log‑average of these normalized metrics offers a simple, inexpensive, and statistically sound method for research evaluation, applicable even when only limited bibliometric data are available. This work thus opens the door to more equitable and cost‑effective assessment practices in academia and research administration.