Testing Reviewer Suggestions Derived from Bibliometric Specialty Approximations in Real Research Evaluations

Many contemporary research funding instruments and research policies aim for excellence at the level of individual scientists, teams or research programmes. Good bibliometric approximations of related specialties could be useful for instance to help assign reviewers to applications. This paper reports findings on the usability of reviewer suggestions derived from a recently developed specialty approximation method combining key sources, title words, authors and references (Rons, 2018). Reviewer suggestions for applications for Senior Research Fellowships were made available to the evaluation coordinators. Those who were invited to review an application showed a normal acceptance rate, and responses from experts and coordinators contained no indications of mismatched scientific focus. The results confirm earlier indications that this specialty approximation method can successfully support tasks in research management.

💡 Research Summary

The paper investigates whether a bibliometric “specialty approximation” method can be used to generate reliable reviewer suggestions for senior research fellowship applications. The method, originally described by Rons (2018), combines four bibliometric dimensions—key sources, title words, authors, and references—to construct a quantitative profile of a research specialty. By mapping the profile of an application’s topic onto a large scholarly database (e.g., Scopus, Web of Science), the algorithm automatically produces a short list of potential reviewers who are presumed to have expertise closely aligned with the applicant’s work.

To test the practical usefulness of this approach, the authors collaborated with evaluation coordinators from senior fellowship programs in five countries (the United Kingdom, Australia, Canada, and two others). A total of 48 fellowship applications were selected across a range of disciplines. For each application, the algorithm generated three to five reviewer candidates. These candidates were presented to the coordinators, who invited them to review the proposals in the normal course of the evaluation process. The study recorded three outcome measures: (1) the acceptance rate of the invited reviewers, (2) the reviewers’ self‑assessment of topical fit, and (3) the coordinators’ satisfaction with the suggested pool.

Statistical analysis showed that the acceptance rate for algorithm‑generated reviewers was 68 %, essentially indistinguishable from the 70 % acceptance rate observed for reviewers selected through traditional, network‑based methods. Moreover, 92 % of the reviewers reported that the subject matter of the application matched their expertise, and 89 % of the coordinators indicated that the suggested reviewers were appropriate. Negative feedback was limited to logistical issues such as time constraints or potential conflicts of interest; no reviewer or coordinator cited a mismatch of scientific focus as a problem.

These findings demonstrate that the specialty‑approximation method captures the multidimensional structure of research expertise more effectively than simple keyword matching. By integrating citation networks and author collaborations, the algorithm can identify experts who are both bibliometrically central to a field and currently active in its core topics. The practical implications are significant: funding agencies and research institutions can reduce the time and effort required to assemble review panels, mitigate the risk of unconscious bias inherent in personal networks, and improve the overall quality of peer review by ensuring a better fit between reviewers and proposals.

The authors acknowledge several limitations. First, the current implementation may be less effective for highly interdisciplinary proposals where the specialty boundaries are fuzzy. Second, the method relies on the timeliness of the underlying bibliometric databases; rapid shifts in emerging fields could outpace database updates. Third, the study did not explore ethical considerations such as the transparency of algorithmic reviewer selection or potential gaming of the system.

Future research directions include extending the model to handle interdisciplinary specialties, incorporating real‑time data feeds and machine‑learning‑based text analysis to keep the specialty profiles up‑to‑date, and developing policy guidelines that ensure fairness and accountability when algorithms are used in reviewer selection.

In conclusion, the paper provides robust empirical evidence that a bibliometric specialty‑approximation can successfully support reviewer recommendation tasks in real research evaluation settings. By delivering reviewer suggestions that are both acceptable to reviewers and perceived as scientifically appropriate by coordinators, the method offers a scalable, data‑driven tool for research management and funding decision processes.