Misconception Diagnosis From Student-Tutor Dialogue: Generate, Retrieve, Rerank

Timely and accurate identification of student misconceptions is key to improving learning outcomes and pre-empting the compounding of student errors. However, this task is highly dependent on the effort and intuition of the teacher. In this work, we present a novel approach for detecting misconceptions from student-tutor dialogues using large language models (LLMs). First, we use a fine-tuned LLM to generate plausible misconceptions, and then retrieve the most promising candidates among these using embedding similarity with the input dialogue. These candidates are then assessed and re-ranked by another fine-tuned LLM to improve misconception relevance. Empirically, we evaluate our system on real dialogues from an educational tutoring platform. We consider multiple base LLM models including LLaMA, Qwen and Claude on zero-shot and fine-tuned settings. We find that our approach improves predictive performance over baseline models and that fine-tuning improves both generated misconception quality and can outperform larger closed-source models. Finally, we conduct ablation studies to both validate the importance of our generation and reranking steps on misconception generation quality.

💡 Research Summary

This paper tackles the problem of automatically diagnosing student misconceptions from student‑tutor dialogues, a task traditionally reliant on teacher intuition and labor‑intensive labeling. The authors propose a three‑stage pipeline—Generate, Retrieve, Rerank—that leverages large language models (LLMs) and dense retrieval to produce high‑quality predictions.

In the Generation stage, a LoRA‑fine‑tuned LLM (either LLaMA‑3.2‑3B Instruct or Qwen‑2.5‑7B Instruct) receives the math question, the student’s selected answer, and the full dialogue, and outputs a natural‑language hypothesis describing a plausible misconception. LoRA adapts only about 0.5 % of the model parameters, enabling efficient domain‑specific tuning with limited data.

The Retrieval stage embeds both the generated hypothesis and all 546 ground‑truth misconception labels using the off‑the‑shelf MiniLM‑L6‑v2 encoder. Cosine similarity ranks the labels, and the top‑k candidates (e.g., k = 10) are passed forward. This dense retrieval step bridges the gap between the free‑form hypothesis and the canonical label taxonomy, providing a scalable semantic matching mechanism.

In the Rerank stage, a second LoRA‑fine‑tuned LLM re‑evaluates the top‑k candidates, producing a final ranking that captures subtle semantic nuances that pure embedding similarity may miss. The highest‑ranked label is taken as the model’s prediction.

The authors construct a new dataset of 922 real student‑tutor conversations collected from an online learning platform, each annotated by expert tutors with a likelihood score for 546 unique misconceptions. Only high‑confidence (≥ 75 %) annotations are retained, ensuring a clear ground truth. The split is performed on unique misconception labels (70 % train, 10 % validation, 20 % test) to assess generalization to unseen misconceptions.

Evaluation uses MAP@k, NDCG, and Recall@k, focusing on the model’s ability to rank the correct misconception near the top. Baselines include: (1) Direct Embedding Matching (no generation), (2) Zero‑shot Claude Sonnet 4.5 classification over the full label set, and (3) TF‑IDF keyword matching. Results show that the full Generate‑Retrieve‑Rerank pipeline consistently outperforms all baselines across metrics, especially in top‑1 precision (MAP@1) and overall ranking quality (NDCG).

Ablation studies confirm the contribution of each component: removing generation degrades performance to the level of direct embedding; omitting reranking reduces NDCG by over 20 %; and eliminating both collapses the system to baseline levels.

Comparisons between open‑source and closed‑source models reveal that LoRA‑fine‑tuned open‑source LLMs can surpass the larger proprietary Claude model, offering comparable or better accuracy while avoiding API costs and privacy concerns.

In summary, the paper demonstrates that a modular pipeline combining LLM‑based hypothesis generation, dense semantic retrieval, and LLM‑driven reranking provides a robust, data‑efficient solution for misconception diagnosis in educational dialogues. The approach is scalable, respects privacy, and offers a clear path toward real‑time, teacher‑assistive tutoring systems.