Learning to Judge: LLMs Designing and Applying Evaluation Rubrics

Large language models (LLMs) are increasingly used as evaluators for natural language generation, applying human-defined rubrics to assess system outputs. However, human rubrics are often static and misaligned with how models internally represent language quality. We introduce GER-Eval (Generating Evaluation Rubrics for Evaluation) to investigate whether LLMs can design and apply their own evaluation rubrics. We evaluate the semantic coherence and scoring reliability of LLM-defined criteria and their alignment with human criteria. LLMs reliably generate interpretable and task-aware evaluation dimensions and apply them consistently within models, but their scoring reliability degrades in factual and knowledge-intensive settings. Closed-source models such as GPT-4o achieve higher agreement and cross-model generalization than open-weight models such as Llama. Our findings position evaluation as a learned linguistic capability of LLMs, consistent within models but fragmented across them, and call for new methods that jointly model human and LLM evaluative language to improve reliability and interpretability.

💡 Research Summary

The paper addresses a growing trend in natural language generation (NLG) research: using large language models (LLMs) as evaluators. Traditionally, LLMs are prompted with static, human‑crafted rubrics that define quality dimensions such as fluency, coherence, and factuality. However, these fixed rubrics may not align with the internal representations that LLMs use to judge language, leading to inconsistencies, bias, and limited adaptability. To investigate whether LLMs can autonomously design and apply their own evaluation criteria, the authors introduce GER‑Eval (Generating Evaluation Rubrics for Evaluation), a two‑stage diagnostic framework.

In the first stage, a model receives a task description and a prompting condition (task‑only, task + contexts, or task + contrastive examples) and is asked to generate a set of evaluation criteria. Each criterion consists of a name, a textual definition, a scoring scale (numeric or categorical), and a short instruction describing how to apply the scale. The second stage uses the generated rubric to score candidate outputs. The scoring can be performed zero‑shot (rubric only) or few‑shot (rubric plus a few demonstration examples). By separating generation from application, the framework enables controlled analysis of (i) how LLMs conceptualize evaluation and (ii) how consistently they apply the resulting rubric.

The authors evaluate GER‑Eval across four established NLG benchmarks—USR (dialogue), SummEval (summarization), SumPubMed (biomedical summarization), and HelpSteer2 (instruction‑following)—and five models: two closed‑source systems (GPT‑4o and GPT‑4o‑mini) and three open‑source models (Mixtral‑8x22B, Llama‑3.3‑70B, Qwen2.5‑72B). Human annotations for each benchmark provide multiple quality dimensions, allowing direct correlation between model‑generated scores and human judgments.

Key Findings

1. Rubric Generation Capability

   • All models can produce 5‑8 structured criteria per task.
   • GPT‑4o and GPT‑4o‑mini generate highly diverse rubrics, with >90 % of criteria being unique across runs.
   • Alignment with human rubrics (the proportion of generated criteria that map to a human‑defined dimension) exceeds 80 % for most datasets, reaching 100 % on dialogue and instruction‑following tasks.
   • Biomedical summarization (SumPubMed) shows the lowest alignment (≈60 %) because specialized terminology and factual precision are harder for models to infer from the task description alone.
   • Prompting with contexts or contrastive examples increases the number of generated criteria and their specificity, but the effect on alignment varies by model (e.g., Mixtral benefits most from task‑only prompts, while Llama improves with contrastive examples).

2. Rubric Application Consistency

   • Within‑model consistency (Pearson correlation between repeated scoring runs) is highest for GPT‑4o (≈0.78–0.84) and lowest for open‑source models (≈0.55–0.68).
   • Correlation with human scores ranges from 0.45 to 0.62 overall, but drops dramatically (<0.30) on knowledge‑intensive tasks such as SumPubMed, indicating that self‑generated rubrics struggle to capture factuality.
   • Cross‑model agreement is modest (average ≈0.42), confirming that evaluation behavior is model‑dependent.

3. Closed‑Source vs. Open‑Source

   • GPT‑4o family consistently outperforms open‑source counterparts in both rubric generation quality and scoring reliability, likely due to larger training corpora and dedicated alignment phases.
   • Mixtral shows competitive alignment on some tasks but lags in internal consistency.
   • Llama and Qwen, despite large parameter counts, exhibit higher redundancy in generated rubrics and lower human alignment, suggesting that alignment data and fine‑tuning are crucial.

4. Failure Modes

   • In factual or knowledge‑heavy settings, the generated rubrics often omit or under‑weight factuality, leading to unreliable scores.
   • The two‑stage pipeline does not provide feedback from the scoring stage back to rubric generation, so poorly performing rubrics are not automatically refined.
   • Model‑specific biases (e.g., verbosity, position effects) persist even when using self‑generated rubrics.

Implications and Future Directions

The study positions evaluation as a learned linguistic capability of LLMs rather than a static external tool. While LLMs can autonomously articulate coherent, task‑aware quality dimensions, the fragmentation across model families limits the universality of such rubrics. To move toward reliable, interpretable, and human‑aligned evaluation, the authors suggest:

• Joint Modeling: Simultaneously train a model to generate rubrics and to score, allowing gradient‑based refinement of criteria based on scoring performance.
• Meta‑Evaluation: Employ a separate “meta‑evaluator” LLM to assess the quality of generated rubrics, creating a feedback loop.
• Hybrid Human‑LLM Rubrics: Combine human‑defined anchors with model‑generated extensions, especially for domains requiring expert knowledge.
• Knowledge‑Augmented Scoring: Integrate external fact‑checking or retrieval modules when applying rubrics to factual tasks.

In conclusion, GER‑Eval demonstrates that LLMs are capable of designing and applying their own evaluation rubrics with reasonable internal consistency, but cross‑model agreement and factual reliability remain open challenges. Closed‑source models currently provide the most trustworthy self‑evaluation, while open‑source models require additional alignment and knowledge‑integration strategies. The work opens a promising research avenue: treating evaluation as an emergent, model‑specific skill that can be studied, improved, and possibly standardized across the AI community.