Peer reviewers equally critique theory, method, and writing, with limited effect on manuscripts' content

Peer review aims to detect flaws and deficiencies in the design and interpretation of studies, and ensure the clarity and quality of their presentation. However, it has been questioned whether peer review fulfils this function. Studies have highlighted a stronger focus of reviewers on critiquing methodological aspects of studies and the quality of writing in biomedical sciences, with less focus on theoretical grounding. In contrast, reviewers in the social sciences appear more concerned with theoretical underpinnings. These studies also found the effect of peer review on manuscripts’ content to be variable, but generally modest and positive. I qualitatively analysed 1,430 peer reviewers’ comments for a sample of 40 social science preprint-publication pairs to identify the key foci of reviewers’ comments. I then quantified the effect of peer review on manuscripts by examining differences between the preprint and published versions using the normalised Levenshtein distance, cosine similarity, and word count ratios for titles, abstracts, document sections and full-texts. I also examined changes in references used between versions and linked changes to reviewers’ comments. Reviewers’ comments were nearly equally split between issues of methodology (30.7%), theory (30.0%), and writing quality (29.2%). Titles, abstracts, and the semantic content of documents remained similar, although publications were typically longer than preprints. Two-thirds of citations were unchanged, 20.9% were added during review and 13.1% were removed. These findings indicate reviewers equally attended to the theoretical and methodological details and communication style of manuscripts, although the effect on quantitative measures of the manuscripts was limited.

💡 Research Summary

The paper investigates how peer reviewers allocate their attention among theoretical grounding, methodological rigor, and writing quality, and how this attention translates into concrete changes in manuscripts. The author selected a sample of 40 social‑science preprint–publication pairs and extracted a total of 1,430 reviewer comments. Each comment was coded into three mutually exclusive categories: methodology, theory, and writing (including style, structure, and grammar). The distribution was strikingly balanced—30.7 % of comments addressed methodological issues, 30.0 % focused on theoretical aspects, and 29.2 % concerned writing quality. This finding challenges earlier claims that reviewers in the biomedical sciences concentrate primarily on methods while social‑science reviewers emphasize theory; instead, social‑science reviewers appear to evaluate all three dimensions with roughly equal intensity.

To quantify the impact of peer review on the manuscripts, the author compared the preprint and the final published version using three complementary textual similarity measures. The normalized Levenshtein distance, which captures the proportion of edit operations needed to transform one text into another, was low (0.07–0.12) across titles, abstracts, and full texts, indicating minimal character‑level alterations. Cosine similarity of TF‑IDF vectors yielded values above 0.94, confirming that the semantic content of the documents remained highly consistent. Word‑count ratios showed that published articles were on average 12 % longer than their preprint counterparts, a growth largely attributable to added tables, figures, supplementary material, and expanded reference lists rather than substantive rewrites of the core narrative.

Reference analysis revealed that two‑thirds (66 %) of citations were unchanged between versions. However, 20.9 % of the references in the final version were newly added, and 13.1 % were removed. The newly introduced citations tended to be recent studies or theoretical works that reviewers explicitly suggested in their “theory” comments, whereas the removed citations were often redundant or tangential. By linking specific reviewer remarks to reference changes, the study demonstrates a clear pathway through which peer feedback can shape the scholarly context of a paper, even when the main text remains largely stable.

Overall, the results paint a nuanced picture of the peer‑review process. Reviewers do indeed devote comparable effort to evaluating theory, methodology, and writing, contradicting the notion of a single dominant focus. Yet the measurable effect of this balanced scrutiny on the manuscript’s textual content is modest. The core arguments, hypotheses, and conclusions tend to survive the review cycle unchanged; most of the observable modifications pertain to lengthening the manuscript and refining its citation network. This suggests that peer review functions more as a quality‑control and polishing mechanism than as a catalyst for major conceptual revisions in social‑science research.

The study’s methodological approach—combining qualitative coding of reviewer feedback with quantitative text‑similarity metrics and citation tracking—offers a replicable framework for future investigations across disciplines. It also raises practical implications: if the goal of peer review is to improve substantive scientific contribution, editorial policies might need to encourage more targeted, content‑driving feedback rather than predominantly stylistic or methodological polishing. Training reviewers to articulate how their suggestions could reshape theoretical framing or empirical interpretation may increase the likelihood that peer review leads to deeper, more meaningful changes in scholarly work.