What Is Novel? A Knowledge-Driven Framework for Bias-Aware Literature Originality Evaluation

Assessing research novelty is a core yet highly subjective aspect of peer review, typically based on implicit judgment and incomplete comparison to prior work. We introduce a literature-aware novelty assessment framework that explicitly learns how humans judge novelty from peer-review reports and grounds these judgments in structured comparison to existing research. Using nearly 80K novelty-annotated reviews from top-tier AI conferences, we fine-tune a large language model to capture reviewer-aligned novelty evaluation behavior. For a given manuscript, the system extracts structured representations of its ideas, methods, and claims, retrieves semantically related papers, and constructs a similarity graph that enables fine-grained, concept-level comparison to prior work. Conditioning on this structured evidence, the model produces calibrated novelty scores and human-like explanatory assessments, reducing overestimation and improving consistency relative to existing approaches.

💡 Research Summary

The paper introduces a literature‑aware, bias‑aware framework for automatically assessing the novelty of scientific manuscripts. Leveraging a newly constructed benchmark of 79,973 peer‑review reports from top AI conferences (NeurIPS and ICLR), the authors extract novelty‑related statements, aggregate them per paper, and assign a normalized novelty label ranging from –1 (not novel) to 2 (highly novel). This dataset captures real reviewer judgments and serves as supervision for fine‑tuning a large language model (LLM).

The system operates in two stages. First, a knowledge‑extraction module converts a target manuscript into a structured representation Kₘₛ = {ideas, methods, contributions, data, experiments}. Each component is used as a query to the Semantic Scholar API, retrieving up to five semantically related papers. The same extraction pipeline is applied to these candidates, yielding comparable tuples Kᵢ. A similarity graph G(V, E) is built where nodes are the manuscript and retrieved papers; edges encode cosine similarity of embedding‑based representations and a detailed overlap profile at the idea, method, and contribution level. The top‑k most similar papers (P_top) are selected as contextual evidence.

Next, Llama‑3.1‑8B‑Instruct is fine‑tuned on the novelty benchmark. Training instances consist of the full manuscript text, the structured knowledge of P_top, and the target outputs: a calibrated novelty score and a human‑style explanatory paragraph. The instruction format forces paper‑centric judgments rather than reviewer‑specific summaries.

Evaluation on a held‑out set of 500 papers shows that the proposed “Novelty Reviewer” outperforms a range of baselines, including general‑purpose LLMs (GPT‑OSS‑20B, Llama‑3.1‑8B‑Instruct, Mistral‑7B‑Instruct, Qwen2.5‑14B‑Instruct, SciLlama) and domain‑adapted reviewers (Paper Reviewer, OpenReviewer). The model achieves the highest accuracy (0.62), F1 (0.323), and Pearson correlation (0.760) with human labels, and it uniquely captures low‑novelty cases, avoiding the common over‑estimation bias of other systems.

A targeted case study on idea‑level plagiarism (paraphrased abstracts with identical concepts) demonstrates that the framework can correctly flag low novelty and cite the original work, whereas most baselines assign high novelty scores and fail to retrieve the source. This underscores the value of structured knowledge extraction and graph‑based comparison for deep semantic matching beyond surface text similarity.

Limitations include dependence on the coverage of external scholarly databases and the fact that the model is trained exclusively on AI/ML conference reviews, which may restrict generalization to other domains. Ethical considerations stress that the tool is intended to assist, not replace, human reviewers, and that the benchmark uses only publicly available review data, avoiding privacy concerns.

In sum, the paper presents a novel, human‑aligned approach that combines large‑scale reviewer supervision with explicit literature grounding, delivering calibrated novelty scores and transparent, evidence‑based explanations that improve consistency and reliability in research novelty assessment.