DistillNote: Toward a Functional Evaluation Framework of LLM-Generated Clinical Note Summaries

Large language models (LLMs) are increasingly used to generate summaries from clinical notes. However, their ability to preserve essential diagnostic information remains underexplored, which could lead to serious risks for patient care. This study introduces DistillNote, an evaluation framework for LLM summaries that targets their functional utility by applying the generated summary downstream in a complex clinical prediction task, explicitly quantifying how much prediction signal is retained. We generated over 192,000 LLM summaries from MIMIC-IV clinical notes with increasing compression rates: standard, section-wise, and distilled section-wise. Heart failure diagnosis was chosen as the prediction task, as it requires integrating a wide range of clinical signals. LLMs were fine-tuned on both the original notes and their summaries, and their diagnostic performance was compared using the AUROC metric. We contrasted DistillNote’s results with evaluations from LLM-as-judge and clinicians, assessing consistency across different evaluation methods. Summaries generated by LLMs maintained a strong level of heart failure diagnostic signal despite substantial compression. Models trained on the most condensed summaries (about 20 times smaller) achieved an AUROC of 0.92, compared to 0.94 with the original note baseline (97 percent retention). Functional evaluation provided a new lens for medical summary assessment, emphasizing clinical utility as a key dimension of quality. DistillNote introduces a new scalable, task-based method for assessing the functional utility of LLM-generated clinical summaries. Our results detail compression-to-performance tradeoffs from LLM clinical summarization for the first time. The framework is designed to be adaptable to other prediction tasks and clinical domains, aiding data-driven decisions about deploying LLM summarizers in real-world healthcare settings.

💡 Research Summary

DistillNote introduces a functional evaluation framework for clinical note summarization by large language models (LLMs), shifting the focus from traditional lexical or semantic similarity metrics to the preservation of diagnostic signal in downstream clinical tasks. Using the MIMIC‑IV database, the authors extracted 64,734 admission notes and generated more than 192,000 summaries with three state‑of‑the‑art LLMs (DeepSeek‑R1‑70B, OpenBioLLM‑70B, and Phi‑4‑14B). Summaries were produced via three strategies: (1) One‑step, a single‑prompt summary of the entire note; (2) Structured, four separate prompts targeting chief complaint, medical history, exam findings, and social/family background, concatenated into a section‑wise summary; and (3) Distilled, a second‑level compression applied to the Structured output. The average compression rates were 36 % (One‑step), 53 % (Structured), and 79 % (Distilled), corresponding to reductions of up to 20‑fold in token count.

To assess functional utility, the authors fine‑tuned each LLM on a binary heart‑failure prediction task using either the full note or one of the three summary types as input. Model performance was measured primarily by AUROC, with AUPRC and F1‑score as secondary metrics. The full‑note baseline achieved AUROC = 0.939, AUPRC = 0.842. One‑step summaries yielded AUROC ≈ 0.926–0.929 (≈1.0 % loss), Structured summaries AUROC ≈ 0.916–0.926 (≈1.3 % loss), and Distilled summaries AUROC ≈ 0.911–0.917 (≈2.2 % loss). Even the most compressed Distilled summaries (≈87 words on average) retained 97 % of the diagnostic signal, demonstrating that substantial text reduction does not catastrophically impair predictive power.

Beyond functional metrics, the study incorporated two orthogonal quality assessments. An “LLM‑as‑judge” framework used Phi‑4 to score summaries on Relevance, Factual Fabrication, and Clinical Actionability on a 1‑5 scale. One‑step summaries received the highest overall scores (3.93 ± 0.29), while Distilled summaries excelled in factuality (3.92 ± 0.26). Statistical testing (ANOVA, Tukey HSD) confirmed significant differences (p < 0.01) and medium‑to‑large effect sizes between strategies. A blinded pairwise comparison by two board‑certified clinicians on 18 cases showed a moderate positive correlation with the LLM‑judge scores (Spearman ρ = 0.67, p < 0.05). Clinicians preferred One‑step summaries overall but noted that Distilled summaries were concise yet sufficient for clear cases, highlighting complementary strengths.

The authors discuss several implications. First, functional evaluation provides a scalable, task‑driven metric that directly reflects clinical safety, addressing a gap in current summarization assessment practices. Second, the compression‑to‑performance trade‑off curves suggest that LLM‑generated summaries could replace full notes in real‑time risk stratification pipelines, reducing computational load and clinician reading time without sacrificing accuracy. Third, the framework is modular and can be extended to other prediction tasks, disease domains, or multi‑label settings.

Limitations include the focus on a single downstream task (heart‑failure prediction), lack of external validation on other institutions, and limited analysis of subtle hallucinations that may persist despite high AUROC. Moreover, detailed hyper‑parameter settings and data splits are not fully disclosed, which could affect reproducibility. Future work should explore broader clinical outcomes, incorporate rigorous factuality verification, and examine the interaction between summary length, model size, and domain‑specific pre‑training.

In conclusion, DistillNote offers a novel, empirically validated methodology for assessing the functional utility of LLM‑generated clinical summaries. By quantifying how much diagnostic information survives compression, it equips healthcare AI developers and policymakers with a concrete, outcome‑oriented benchmark for safe deployment of summarization models in real‑world clinical environments.