Modelling and Classifying the Components of a Literature Review

Previous work has demonstrated that AI methods for analysing scientific literature benefit significantly from annotating sentences in papers according to their rhetorical roles, such as research gaps, results, limitations, extensions of existing methodologies, and others. Such representations also have the potential to support the development of a new generation of systems capable of producing high-quality literature reviews. However, achieving this goal requires the definition of a relevant annotation schema and effective strategies for large-scale annotation of the literature. This paper addresses these challenges in two ways: 1) it introduces a novel, unambiguous annotation schema that is explicitly designed for reliable automatic processing, and 2) it presents a comprehensive evaluation of a wide range of large language models (LLMs) on the task of classifying rhetorical roles according to this schema. To this end, we also present Sci-Sentence, a novel multidisciplinary benchmark comprising 700 sentences manually annotated by domain experts and 2,240 sentences automatically labelled using LLMs. We evaluate 37 LLMs on this benchmark, spanning diverse model families and sizes, using both zero-shot learning and fine-tuning approaches. The experiments reveal that modern LLMs achieve strong results on this task when fine-tuned on high-quality data, surpassing 96% F1, with both large proprietary models such as GPT-4o and lightweight open-source alternatives performing well. Moreover, augmenting the training set with semi-synthetic LLM-generated examples further boosts performance, enabling small encoders to achieve robust results and substantially improving several open decoder models.

💡 Research Summary

The paper tackles the problem of automatically identifying the rhetorical role of sentences in scientific articles, a prerequisite for building systems that can generate high‑quality literature reviews. It makes two principal contributions. First, it proposes a concise, unambiguous annotation schema consisting of seven categories: Overall, Research Gap, Description, Result, Limitation, Extension, and Other. The schema is deliberately designed to avoid the vagueness and overlap that plagued earlier taxonomies (e.g., the 12‑class scheme of Khoo et al., 2011). Detailed definitions and examples are provided for each class, with special attention to the often‑confused Limitation and Description categories, thereby enabling consistent labeling by both human annotators and machine learning models.

Second, the authors construct a new multilingual benchmark called Sci‑Sentence. It contains 700 sentences manually annotated by domain experts across several disciplines (life sciences, computer science, social sciences, etc.) and an additional 2,240 sentences automatically labeled by several existing large language models (LLMs). The dataset is publicly released and serves as a realistic testbed for both zero‑shot and fine‑tuned approaches.

Using Sci‑Sentence, the study evaluates 37 LLMs spanning encoder‑only (BERT, SciBERT), encoder‑decoder (T5, UL2), and decoder‑only (Llama, Mistral, Gemma) families, with model sizes ranging from 1 B to multi‑hundred‑billion parameters. Two learning paradigms are examined: zero‑shot prompting and supervised fine‑tuning, the latter employing LoRA and NEFT adapters. Fine‑tuned models achieve an average F1 above 96 %, confirming that modern LLMs can reach near‑human performance when trained on high‑quality data. Proprietary giants such as GPT‑4o attain the highest scores, yet lightweight open‑source alternatives like SuperNova‑Medius and Nemotron‑8B also surpass 94 % F1, demonstrating that strong performance is not exclusive to commercial systems.

Error analysis reveals that the Limitation and Description classes are the most error‑prone, often being misclassified due to lexical similarity and contextual ambiguity. Decoder‑only models generally obtain the best overall results, but domain‑adapted encoder‑only models (e.g., SciBERT) remain competitive, offering a more resource‑efficient option for large‑scale processing.

A notable finding is the benefit of semi‑synthetic data augmentation. Adding LLM‑generated sentences to the training set improves the performance of small encoders by 3–5 % absolute F1 and yields measurable gains for several open‑source decoders. This suggests that synthetic data can effectively compensate for limited manually labeled resources.

In summary, the paper demonstrates that (1) a well‑crafted, limited‑scope annotation schema enables reliable automatic classification of scientific sentence roles, (2) contemporary LLMs, when fine‑tuned on a curated benchmark, achieve state‑of‑the‑art performance, and (3) synthetic data augmentation is a practical strategy to boost smaller models. Future work may explore further schema refinement, active‑learning labeling pipelines, and integration of the classifier into end‑to‑end literature‑review generation systems.