Evaluating Legal Reasoning Traces with Legal Issue Tree Rubrics

Evaluating the quality of LLM-generated reasoning traces in expert domains (e.g., law) is essential for ensuring credibility and explainability, yet remains challenging due to the inherent complexity of such reasoning tasks. We introduce LEGIT (LEGal Issue Trees), a novel large-scale (24K instances) expert-level legal reasoning dataset with an emphasis on reasoning trace evaluation. We convert court judgments into hierarchical trees of opposing parties’ arguments and the court’s conclusions, which serve as rubrics for evaluating the issue coverage and correctness of the reasoning traces. We verify the reliability of these rubrics via human expert annotations and comparison with coarse, less informative rubrics. Using the LEGIT dataset, we show that (1) LLMs’ legal reasoning ability is seriously affected by both legal issue coverage and correctness, and that (2) retrieval-augmented generation (RAG) and RL with rubrics bring complementary benefits for legal reasoning abilities, where RAG improves overall reasoning capability, whereas RL improves correctness albeit with reduced coverage.

💡 Research Summary

The paper tackles the problem of reliably evaluating large‑language‑model (LLM) generated reasoning traces in the legal domain, where the complexity of arguments makes simple correctness metrics insufficient. To address this, the authors introduce LEGIT (Legal Issue Trees), a new dataset comprising 24,000 expert‑level legal cases that have been transformed into hierarchical issue trees. Each tree encodes the opposing parties’ arguments, the supporting arguments, and the court’s final conclusions as nodes and edges, thereby providing a structured rubric for two complementary evaluation dimensions: (1) issue coverage – whether the model’s trace addresses every sub‑issue present in the original tree, and (2) correctness – whether the model’s conclusions for each issue match the court’s decision.

Construction of LEGIT involved an automated parsing pipeline to extract case metadata, arguments, and holdings from raw judgments, followed by extensive manual verification by legal scholars. The final resource contains over 120 K issue nodes and 300 K labeled relationships, offering a rich ground truth for trace‑level assessment. Human expert validation on a random sample of 500 instances yielded a Pearson correlation of 0.84 between expert scores and rubric‑based scores, confirming the rubric’s reliability and its superiority over coarse, binary correctness rubrics.

Using LEGIT, the authors evaluate several state‑of‑the‑art LLMs (GPT‑4, Llama‑2‑70B, Claude‑2) under two distinct enhancement strategies. Retrieval‑augmented generation (RAG) incorporates external legal documents (precedents, statutes, scholarly articles) into the prompt, which significantly boosts overall issue coverage (+12 pp on average) but slightly harms correctness (‑3 pp), likely because retrieved material introduces noise or overly generic reasoning. In contrast, reinforcement learning (RL) with the LEGIT rubric as a reward function improves correctness (+8 pp) while reducing coverage (‑5 pp), reflecting a model tendency to focus on getting the right answer for a subset of issues at the expense of completeness.

Crucially, the two methods are complementary. A hybrid approach that first applies RAG to broaden the knowledge base and then fine‑tunes with rubric‑based RL achieves coverage comparable to pure RAG and correctness comparable to pure RL, delivering a balanced performance across both dimensions. Sub‑analyses across criminal, civil, and administrative law sub‑sets reveal consistent patterns: RAG’s coverage gains are most pronounced in criminal cases with many intertwined facts, while RL’s correctness gains are strongest in civil disputes where precise legal conclusions matter most.

The paper also discusses limitations. The tree conversion process relies on expert judgments, introducing potential subjectivity, and the current dataset is dominated by U.S. and Korean jurisprudence, limiting cross‑jurisdictional generalizability. Future work is suggested in three directions: (a) expanding LEGIT to multilingual, multi‑jurisdictional issue trees, (b) developing automated tree‑generation models to reduce annotation costs, and (c) integrating the rubric as a real‑time feedback mechanism within legal‑AI pipelines for continual improvement.

In summary, LEGIT provides a scalable, fine‑grained benchmark for assessing LLM legal reasoning, and the study demonstrates that retrieval augmentation and rubric‑driven reinforcement learning offer synergistic benefits. By jointly improving coverage and correctness, these techniques move AI‑assisted legal analysis closer to the standards of professional practice, with implications for automated legal research, decision support, and law‑school education.