Testing bibliometric indicators by their prediction of scientists promotions

We have developed a method to obtain robust quantitative bibliometric indicators for several thousand scientists. This allows us to study the dependence of bibliometric indicators (such as number of publications, number of citations, Hirsch index…) on the age, position, etc. of CNRS scientists. Our data suggests that the normalized h index (h divided by the career length) is not constant for scientists with the same productivity but differents ages. We also compare the predictions of several bibliometric indicators on the promotions of about 600 CNRS researchers. Contrary to previous publications, our study encompasses most disciplines, and shows that no single indicator is the best predictor for all disciplines. Overall, however, the Hirsch index h provides the least bad correlations, followed by the number of papers published. It is important to realize however that even h is able to recover only half of the actual promotions. The number of citations or the mean number of citations per paper are definitely not good predictors of promotion.

💡 Research Summary

The paper presents a large‑scale empirical investigation of how bibliometric indicators relate to career advancement within the French National Centre for Scientific Research (CNRS). By integrating publication and citation data from Web of Science/Scopus with internal personnel records, the authors constructed a robust database covering several thousand scientists across a wide spectrum of disciplines (physics, chemistry, life sciences, engineering, social sciences, humanities, etc.). For each researcher they extracted core metrics – total number of papers (P), total citations (C), Hirsch index (h), and average citations per paper (C/P) – and paired these with demographic variables such as age, career start year, and current rank (researcher, senior researcher, director, etc.).

The first analytical step examined the dependence of these metrics on age and career length. While raw counts of papers and citations increase monotonically with seniority, the normalized h‑index (h divided by career length) shows a systematic decline for older scientists who otherwise have comparable annual productivity. This suggests that citation accumulation slows down over time, possibly due to changes in field citation practices or diminishing marginal impact of later work. The authors also highlight pronounced disciplinary differences: fields with fast‑moving citation cultures (e.g., physics, chemistry) display higher average citations, whereas in the social sciences the sheer volume of publications appears more influential.

To assess predictive power, the authors treated promotion (e.g., from researcher to senior researcher) as a binary outcome and fitted logistic regression models using each bibliometric indicator separately. Model performance was evaluated with the area under the ROC curve (AUC), overall accuracy, and F1‑score, employing five‑fold cross‑validation to guard against over‑fitting. Across the full sample of roughly 600 promotion events, the h‑index consistently achieved the highest AUC (≈0.68) and accuracy (≈0.62). The number of papers followed closely (AUC ≈0.62, accuracy ≈0.58). In contrast, total citations and mean citations per paper performed near chance level (AUC ≤0.55), indicating that sheer citation volume does not translate into promotion likelihood.

When the analysis was stratified by discipline, the superiority of any single metric vanished. In the natural sciences, h remained the strongest predictor, whereas in the social sciences and humanities the count of publications carried more weight. This heterogeneity underscores that promotion committees implicitly value different aspects of scholarly output depending on disciplinary norms. Moreover, even the best‑performing indicator (h) correctly identified only about half of the actual promotions, revealing that bibliometrics capture only a portion of the criteria used in personnel decisions.

The authors discuss several limitations. First, the normalization of h by career length is simplistic; more sophisticated models could account for non‑linear citation accrual and cohort effects. Second, the data lack qualitative dimensions such as leadership in large projects, mentorship, external grant acquisition, and peer‑review assessments, all of which are known to influence promotion outcomes. Third, the study is confined to a single national research system, which may limit generalizability to other institutional contexts.

In response, the paper proposes a hybrid evaluation framework that combines quantitative bibliometrics with structured qualitative assessments (e.g., peer‑review scores, project leadership metrics, funding records). It also calls for the development of discipline‑specific bibliometric composites that respect the distinct publication and citation cultures of each field.

In conclusion, the study provides robust evidence that among the traditional bibliometric measures, the Hirsch index offers the most reliable, though still modest, predictor of promotion within the CNRS. No single metric universally outperforms others across all disciplines, and reliance on bibliometrics alone would overlook a substantial portion of the factors that drive career advancement. Policymakers and research managers are therefore advised to treat bibliometric indicators as informative but supplementary inputs within a broader, multi‑dimensional evaluation system.