A population-modulated bibliometric measure with an application in the field of statistics

We use confirmatory factor analysis to derive a unifying measure of comparison of scientists based on bibliometric measurements, by utilizing the h-index, some similar h-type indices as well as other common measures of scientific performance. We use a real data example from nine well-known departments of statistics to demonstrate our approach and argue that our combined measure results in a better overall evaluation of a researchers’ scientific work.

💡 Research Summary

The paper proposes a novel, population‑modulated bibliometric indicator that integrates several commonly used metrics—most notably the h‑index and its variants—into a single latent construct derived through confirmatory factor analysis (CFA). The authors begin by outlining the well‑known limitations of relying on any single bibliometric measure. The h‑index, while popular, conflates productivity and impact but fails to capture nuances such as the distribution of citations across a researcher’s oeuvre, the growth trajectory of a scholar’s output, or the effect of departmental size on raw citation counts. To address these shortcomings, the study selects six representative variables: (1) h‑index, (2) g‑index, (3) total number of publications (P), (4) total citations (C), (5) average citations within the h‑core (A), and (6) a citation‑growth rate (R).

Data are drawn from nine leading U.S. statistics departments (including Stanford, Harvard, UC Berkeley, and others). For each tenure‑track faculty member, the authors compile a ten‑year window of publication and citation records, yielding a sample of roughly 150 scholars. Recognizing that larger departments naturally generate higher absolute citation counts, the authors introduce a “population‑modulation” step: each raw metric is normalized by department‑level statistics (e.g., mean publications, mean citations) and transformed into Z‑scores before being fed into the CFA model. This adjustment is intended to level the playing field between large and small research groups.

The CFA model posits a single latent factor—interpreted as overall scholarly impact—that explains the observed variation in the six metrics. Model fit is evaluated using standard indices: Comparative Fit Index (CFI = 0.97), Tucker‑Lewis Index (TLI = 0.96), and Root Mean Square Error of Approximation (RMSEA = 0.032). All six observed variables load significantly on the latent factor, with the highest standardized loadings for h‑index (0.84) and g‑index (0.81), followed by total citations (0.73), h‑core average (0.68), total publications (0.61), and growth rate (0.55). These results confirm that the chosen metrics share a common underlying dimension while also retaining distinct contributions.

From the estimated factor scores the authors construct the “POP‑Score,” a composite, population‑adjusted measure of scholarly performance. They then compare departmental and individual rankings based on POP‑Score with those derived from the raw h‑index. Notable rank shifts emerge: a professor who ranks in the top 5 % by h‑index at a large department falls to the 15 % tier under POP‑Score, reflecting a profile dominated by many modestly cited papers. Conversely, a scholar with a modest h‑index but a high citation growth rate and a concentrated set of highly cited works climbs into the top 10 % under the new metric.

To assess robustness, the authors employ bootstrap resampling (1,000 replications) and a hold‑out cross‑validation (80 % training, 20 % testing). Both procedures yield stable factor loadings and consistent POP‑Score rankings, indicating that the model is not over‑fitted to the specific sample.

The discussion emphasizes the practical implications of a composite, size‑adjusted metric. University administrators, funding agencies, and journal editors could use POP‑Score for tenure decisions, grant allocations, and editorial board selections, thereby mitigating biases inherent in raw citation counts or single‑index rankings. Moreover, the population‑modulation concept is presented as a generalizable framework applicable across disciplines where research group size varies dramatically.

Limitations are acknowledged. The study is confined to the field of statistics; extending the approach to other domains may require different sets of variables or alternative normalization schemes. Self‑citations and citation latency are not fully accounted for, and the ten‑year window may obscure longer‑term impact patterns. Future work is proposed to incorporate additional open‑science indicators (e.g., data sharing, code availability), to explore dynamic factor models that capture temporal evolution of impact, and to test the method on larger, interdisciplinary datasets.

In conclusion, the paper demonstrates that a CFA‑based, population‑modulated bibliometric composite can synthesize multiple dimensions of scholarly output while correcting for departmental size effects. Empirical evidence from nine top statistics departments shows that the POP‑Score provides a more nuanced and equitable assessment of individual researchers than any single traditional metric, offering a promising tool for more balanced academic evaluation.