Beyond Holistic Scores: Automatic Trait-Based Quality Scoring of Argumentative Essays

Automated Essay Scoring systems have traditionally focused on holistic scores, limiting their pedagogical usefulness, especially in the case of complex essay genres such as argumentative writing. In educational contexts, teachers and learners require interpretable, trait-level feedback that aligns with instructional goals and established rubrics. In this paper, we study trait-based Automatic Argumentative Essay Scoring using two complementary modeling paradigms designed for realistic educational deployment: (1) structured in-context learning with small open-source LLMs, and (2) a supervised, encoder-based BigBird model with a CORAL-style ordinal regression formulation, optimized for long-sequence understanding. We conduct a systematic evaluation on the ASAP++ dataset, which includes essay scores across five quality traits, offering strong coverage of core argumentation dimensions. LLMs are prompted with designed, rubric-aligned in-context examples, along with feedback and confidence requests, while we explicitly model ordinality in scores with the BigBird model via the rank-consistent CORAL framework. Our results show that explicitly modeling score ordinality substantially improves agreement with human raters across all traits, outperforming LLMs and nominal classification and regression-based baselines. This finding reinforces the importance of aligning model objectives with rubric semantics for educational assessment. At the same time, small open-source LLMs achieve a competitive performance without task-specific fine-tuning, particularly for reasoning-oriented traits, while enabling transparent, privacy-preserving, and locally deployable assessment scenarios. Our findings provide methodological, modeling, and practical insights for the design of AI-based educational systems that aim to deliver interpretable, rubric-aligned feedback for argumentative writing.

💡 Research Summary

This paper addresses a critical gap in automated essay scoring (AES) by moving beyond holistic scores toward trait‑based assessment for argumentative writing. Recognizing that holistic scoring obscures the multidimensional nature of writing quality and offers limited formative feedback, the authors focus on five rubric‑defined traits: Ideas & Content, Organization, Word Choice, Sentence Fluency, and Conventions. They evaluate two complementary modeling paradigms under realistic classroom constraints.

The first paradigm employs small open‑source large language models (LLMs) – Llama‑3.1 (8 B), Gemma‑3 (12 B), and Ministral‑3 (8 B) – using a structured in‑context learning approach. Prompts are carefully crafted to include role specification (teacher), explicit trait definitions, rubric‑based scoring criteria, exemplar essays with scores, and step‑by‑step reasoning instructions. The models generate both a trait score and a confidence estimate, enabling downstream systems to weight feedback by model certainty. For comparison, the authors also include a small proprietary model (GPT‑4o‑mini) and the state‑of‑the‑art GPT‑5.1, highlighting cost, privacy, and transparency differences.

The second paradigm is a supervised encoder‑based model built on BigBird, a long‑document transformer capable of processing up to 4096 tokens. To respect the ordinal nature of rubric scores, the authors adopt the CORAL (Consistent Rank Logits) formulation, which models score thresholds rather than independent class labels. They collapse the original six‑point scale into three semantically meaningful categories (weak, fair, strong), improving interpretability and reducing artificial precision. The BigBird‑CORAL model therefore aligns directly with the ordered structure of educational rubrics.

Experiments are conducted on the argumentative subset of the ASAP++ dataset (1,783 Grade‑8 essays). Each essay is annotated on the five traits using a six‑point scale, which the authors map to the three ordinal categories. Evaluation metrics include Quadratic Weighted Kappa (QWK) and Pearson correlation. Results show that the BigBird‑CORAL model consistently outperforms all baselines across every trait, achieving QWK scores above 0.78 for Ideas & Content and Organization, indicating strong agreement with human raters. Among LLMs, the reasoning‑oriented Ministral‑3 yields the best performance on language‑centric traits (Word Choice, Sentence Fluency), while Llama‑3.1 and Gemma‑3 perform competitively but slightly lower. Confidence estimates produced by the LLMs correlate with human judgments (≈0.65), suggesting they can be used to filter or weight automated feedback.

Key insights emerge: (1) Explicitly modeling the ordinal structure of rubric scores dramatically improves alignment with human grading practices; (2) Long‑sequence transformers like BigBird capture discourse‑level argumentative structure essential for trait‑level assessment; (3) Small open‑source LLMs, when prompted with well‑designed rubric‑aligned examples, can deliver competitive scores for certain traits without any task‑specific fine‑tuning, offering a privacy‑preserving, transparent alternative to proprietary systems; (4) Providing confidence estimates adds a layer of interpretability useful for educators to gauge the reliability of automated feedback.

The authors conclude that trait‑based automatic argumentative essay scoring is feasible and pedagogically valuable. By combining ordinal regression with long‑document modeling, and by demonstrating the practical utility of open‑source LLMs, the work paves the way for AI‑driven assessment tools that deliver interpretable, rubric‑aligned feedback, support formative learning, and respect institutional constraints on data privacy and model transparency.