Personal connections between creators and evaluators of scientific works are ubiquitous, and the possibility of bias ever-present. Although connections have been shown to bias prospective judgments of (uncertain) future performance, it is unknown whether such biases occur in the much more concrete task of assessing the scientific validity of already completed work, and if so, why. This study presents evidence that personal connections between authors and reviewers of neuroscience manuscripts are associated with biased judgments and explores the mechanisms driving the effect. Using reviews from 7,981 neuroscience manuscripts submitted to the journal PLOS ONE, which instructs reviewers to evaluate manuscripts only on scientific validity, we find that reviewers favored authors close in the co-authorship network by ~0.11 points on a 1.0 - 4.0 scale for each step of proximity. PLOS ONE's validity-focused review and the substantial amount of favoritism shown by distant vs. very distant reviewers, both of whom should have little to gain from nepotism, point to the central role of substantive disagreements between scientists in different "schools of thought." The results suggest that removing bias from peer review cannot be accomplished simply by recusing the closely-connected reviewers, and highlight the value of recruiting reviewers embedded in diverse professional networks.
Around the globe, public and private organizations invest more than $2 trillion into research and development (Industrial Research Institute, 2017). Many of these organizations, including funding agencies and publishers of scientific research, face the challenging task of allocating limited financial or reputational resources across scientific projects, which require increasingly deep and varied domain expertise to evaluate (Jones, 2009;Wuchty, Jones, & Uzzi, 2007). Despite its ubiquity, however, peer review faces persistent critiques of low reliability and bias. Reviewers of a particular scientific work disagree with each other's assessment notoriously often (Bornmann, 2011;Campanario, 1998;Cicchetti, 1991;Marsh, Jayasinghe, & Bond, 2008). Indeed, agreement is often only marginally better than chance and comparable to agreement achieved for Rorschach inkblot tests (Lee, 2012). A bigger concern than the variance of reviews, however, is reviewers' bias for or against particular social and intellectual groups, and particularly those to whom they are personally connected. Given that scientists often work on highly specialized topics in small, dense clusters, the most relevant expert evaluators are typically peers of the research creators and many organizations turn to them for input. Nevertheless, many researchers have suspected that personal connections between reviewers and creators are the locus of nepotism and other systematic biases.
Several studies of scientific evaluation have demonstrated that personal connections are, indeed, associated with biased reviews. For example, recent studies document that reviewers of grant proposals and candidates for promotion favor the research of collaborators and coworkers (Bagues, Sylos-Labini, & Zinovyeva, 2016;Jang, Doh, Kang, & Han, 2016;Sandström & Hällsten, 2007;van den Besselaar, 2012).
Other research documents that higher ratings tend to be given to the research of men (Bagues, Sylos-Labini, & Zinovyeva, 2017;Wennerås & Wold, 1997). These patterns of widespread disagreement and bias in scientific evaluation greatly complicate selection of the most deserving research, which generates a new problem, that of “reviewing the reviewers” to identify which of the cacophonous voices provides unbiased information or de-biasing the reviews. Meanwhile, from the perspective of scientists and scholars, evaluation decisions that drive their careers and billions of research dollars are possibly unfair and, to a large extent, the “luck of the reviewer draw” (Cole, Cole, & Simon, 1981, p. 885).
Despite the centrality of peer review to the scientific enterprise and increasing research attention devoted to it, important questions remain. First, existing studies of reviewer bias have focused on prospective judgments, like promotions and funding competitions. Administrators’ and reviewers’ task in these settings is to predict future performance. In this context, prospective judgments are inherently uncertain and may hinge on information asymmetry, such that particular reviewers have private information about the applicant that other reviewers do not possess. In contrast, it is unknown whether personal connections influence retrospective judgments as in manuscript review, where the task is to assess completed work. Consequently, uncertainty regarding the evaluated work should be lower and, in principle, all reviewers should have equal access to the relevant information, presented within the manuscript. In this way, it is plausible that personal connections are minimally associated with bias in retrospective evaluations.
Second, current studies do not distinguish among mechanism(s) driving bias. Reviewers may favor the work from closely connected authors for many reasons, including nepotism, similar tastes for “soft” evaluation criteria like “significance” or “novelty,” and even shared views on contested substantive matters -a view we call “schools of thought” to suggest deeper shared theoretical, methodological and epistemological assumptions and commitments. Disambiguating these mechanisms is critical because the effectiveness of policies designed to mitigate reviewer bias hinge on the mechanisms driving it. In the case of nepotism, the most effective policy may be to recuse reviewers closely connected with those reviewed or provide reviewer training on conscious and non-conscious biases in judgment. In the case of soft evaluation criteria, it may be important to separate the review process into technical “objective” and softer “subjective” components. With respect to schools of thought, it may be important to select reviewers belonging to diverse schools. In practice, these mechanisms are difficult to disentangle: personal networks overlap with individuals’ scientific views, and evaluations typically collapse technical and soft criteria (Lamont, 2009;Lee, 2012;Travis & Collins, 1991).
This study addresses both aforementioned shortcomings of the literature on scientific evaluati
This content is AI-processed based on open access ArXiv data.
