Language Bottleneck Models for Qualitative Knowledge State Modeling

Accurately assessing student knowledge is central to education. Cognitive Diagnosis (CD) models estimate student proficiency at a fixed point in time, while Knowledge Tracing (KT) methods model evolving knowledge states to predict future performance. However, existing approaches either provide quantitative concept mastery estimates with limited expressivity (CD, probabilistic KT) or prioritize predictive accuracy at the cost of interpretability (deep learning KT). We propose Language Bottleneck Models (LBMs), where an encoder LLM produces textual knowledge state summaries, which a decoder LLM uses to predict future performance. This produces interpretable summaries that can express nuanced insights–such as misconceptions–that CD and KT models cannot capture. Extensive validation across synthetic and real-world datasets shows LBMs reveal qualitative insights beyond what CD and KT models can capture, while achieving competitive accuracy with improved sample efficiency. We demonstrate that the encoder and decoder can be fine-tuned with reinforcement learning and supervised fine-tuning respectively to improve both summary quality and predictive performance.

💡 Research Summary

The paper introduces Language Bottleneck Models (LBMs), a novel two‑stage framework that uses large language models (LLMs) to represent a student’s knowledge state as a concise natural‑language summary rather than a vector of concept mastery scores. In the first stage, an encoder LLM (fθ) ingests a student’s interaction history (questions, concepts, responses, correctness) and generates a textual knowledge‑state summary (Ŝ). This summary becomes the sole intermediate representation. In the second stage, a decoder LLM (gϕ) conditions only on Ŝ and a target question q to predict the probability of a correct answer, p(c=1|q, Ŝ). The authors demonstrate that, given a perfect summary, strong LLMs (e.g., GPT‑4o) can achieve near‑perfect prediction accuracy, indicating that the bottleneck is expressive enough. Consequently, the main challenge is learning faithful summaries. To address this, they train the encoder with reinforcement learning using Group Relative Policy Optimization (GRPO). The encoder produces multiple candidate summaries; each is rewarded based on downstream decoder performance (reconstruction and future‑prediction accuracy), summary length penalty, and optional structural constraints (e.g., inclusion of a “Misconceptions” section). The reward function balances these components with hyperparameters. The decoder is fine‑tuned via supervised learning on summary‑question pairs with binary correctness labels.

Experiments span a synthetic arithmetic benchmark (with programmed mastery and systematic misconceptions) and two real‑world datasets (Eedi and XES3G5M) filtered to approximate static knowledge states. LBMs are evaluated with various open‑source (Qwen‑2.5‑3B/7B, Gemma‑3‑12B/27B) and closed‑source (GPT‑4o‑mini, GPT‑4o, GPT‑5) backbones, and compared against 14 baseline CD and KT models. Results show that LBMs achieve competitive accuracy while requiring far fewer training samples, demonstrating superior sample efficiency. Qualitative evaluation using LLM‑as‑judge metrics and a detailed case study reveals that LBMs can surface nuanced misconceptions (e.g., errors with the operand 6, failures on negative numbers) in human‑readable form—information that traditional CD models only provide as coarse proficiency scores.

The paper also discusses steerability: prompt engineering of the encoder, reward‑based biasing toward human‑preferred summary structures, and post‑generation editing to incorporate teacher observations. These mechanisms enable interactive human‑model workflows, making the approach practical for classroom diagnostics. In sum, LBMs fuse interpretability (through natural‑language bottlenecks) with strong predictive performance, offering a new paradigm for knowledge‑state modeling that bridges the gap between cognitive diagnosis and modern knowledge tracing.