EchoReview: Learning Peer Review from the Echoes of Scientific Citations

As the volume of scientific submissions continues to grow rapidly, traditional peer review systems are facing unprecedented scalability pressures, highlighting the urgent need for automated reviewing methods that are both scalable and reliable. Existing supervised fine-tuning approaches based on real review data are fundamentally constrained by single-source of data as well as the inherent subjectivity and inconsistency of human reviews, limiting their ability to support high-quality automated reviewers. To address these issues, we propose EchoReview, a citation-context-driven data synthesis framework that systematically mines implicit collective evaluative signals from academic citations and transforms scientific community’s long-term judgments into structured review-style data. Based on this pipeline, we construct EchoReview-16K, the first large-scale, cross-conference, and cross-year citation-driven review dataset, and train an automated reviewer, EchoReviewer-7B. Experimental results demonstrate that EchoReviewer-7B can achieve significant and stable improvements on core review dimensions such as evidence support and review comprehensiveness, validating citation context as a robust and effective data paradigm for reliable automated peer review.

💡 Research Summary

The paper addresses the growing scalability challenge of peer review in the face of exploding submission volumes, arguing that existing supervised fine‑tuning (SFT) approaches suffer from two fundamental drawbacks: (1) they rely on a single, narrow source of real review data (mostly AI conferences that release Open‑Review records), which limits cross‑disciplinary generalization, and (2) human reviews are inherently subjective and inconsistent, leading to high inter‑reviewer variance. To overcome these limitations, the authors propose EchoReview, a citation‑context‑driven data synthesis framework that automatically mines implicit evaluative signals from scholarly citations and converts them into structured, review‑style data without any human annotation.

The pipeline consists of four stages. First, high‑impact papers from ACL, EMNLP, ICLR, ICML, and NeurIPS (2020‑2022) are collected. For each paper, the authors retrieve all citing works via the Semantic Scholar API, download the LaTeX source (.tex and .bib), and locate every \cite{key} occurrence. A three‑sentence sliding window (the citing sentence plus its immediate predecessor and successor) is extracted as the citation context.

Second, the raw contexts are processed with GPT‑4o to (a) classify polarity (Strength, Weakness, or Neutral) and re‑phrase them into a standardized review comment, and (b) perform “deep evaluation mining” by asking diagnostic questions that surface implicit endorsements (e.g., method adoption, experimental replication) or hidden criticisms. Only positive and negative citations are kept.

Third, to avoid redundancy, a semantic‑level deduplication step clusters comments that convey the same core insight and retains the most comprehensive formulation.

Fourth, because citation‑derived comments often lack explicit evidence, the framework augments each comment with 1‑3 verbatim evidence passages extracted from the cited paper. These passages, together with the comment, are fed back to GPT‑4o to generate a compact Chain‑of‑Thought (CoT) in an Evidence‑Reasoning‑Conclusion format: each reasoning step starts with a direct quotation, followed by a brief analytical bridge, and ends with a clear Strength or Weakness statement. An independent auditor model (Qwen‑max) then evaluates each CoT for citation validity, logical coherence, and overall explanatory quality; only samples passing a predefined threshold are retained.

The resulting dataset, EchoReview‑16K, contains 16,306 high‑quality review samples, each comprising structured Strength/Weakness lists, associated evidence passages, and a CoT. The authors also construct EchoReview‑Bench, a held‑out test suite (≈1,600 samples) for systematic evaluation.

Using EchoReview‑16K, they fine‑tune a 7‑billion‑parameter LLaMA‑2‑based model (EchoReviewer‑7B) via SFT. Experiments compare EchoReviewer‑7B against recent automated reviewers such as ReviewMT and DeepReviewer on multiple dimensions: Evidence Support, Review Comprehensiveness, Faithfulness, and Alignment with human reviews. EchoReviewer‑7B consistently outperforms baselines, achieving 12‑15 percentage‑point gains on Evidence Support and Comprehensiveness, and demonstrating higher consistency with human judgments.

Key insights include: (1) citation contexts act as a form of collective, long‑term community feedback, mitigating individual reviewer bias; (2) the fully automated pipeline can generate large‑scale, cross‑conference, cross‑year training data without relying on open‑review policies; (3) augmenting citation‑derived comments with explicit evidence and CoT dramatically improves model interpretability and reduces hallucination.

The authors acknowledge limitations: the current study focuses on AI conference papers, and citation practices differ across disciplines (e.g., biomedical or physics literature), which may require domain‑specific adaptation. Moreover, not all citations convey clear evaluative intent, so accurate polarity detection remains a challenge.

In conclusion, EchoReview introduces a novel paradigm for synthesizing peer‑review data from scholarly citations, demonstrates that such data can be transformed into high‑quality, evidence‑grounded training material, and shows that models trained on this data achieve superior automated reviewing performance. This work paves the way for scalable, cross‑disciplinary automated peer review systems that leverage the implicit wisdom embedded in the citation network.