Building Scaffolding Dialogue Data with LLM-Simulated Novices

High-quality, multi-turn instructional dialogues between novices and experts are essential for developing AI systems that support teaching, learning, and decision-making. These dialogues often involve scaffolding – the process by which an expert supports a novice’s thinking through questions, feedback, and step-by-step guidance. However, such data are scarce due to privacy concerns in recording and the vulnerability inherent in help-seeking. We present SimInstruct, a scalable, expert-in-the-loop tool for collecting scaffolding dialogues. Using teaching development coaching as an example domain, SimInstruct simulates novice instructors via LLMs, varying their teaching challenges and LLM’s persona traits, while human experts provide multi-turn feedback, reasoning, and instructional support. This design enables the creation of realistic, pedagogically rich dialogues without requiring real novice participants. Our results reveal that persona traits, such as extroversion and introversion, meaningfully influence how experts engage. Compared to real mentoring recordings, SimInstruct dialogues demonstrate comparable pedagogical relevance and cognitive depth. Experts also reported the process as engaging and reflective, improving both data quality and their own professional insight. We further fine-tuned a LLaMA model to be an expert model using the augmented dataset, which outperformed GPT-4o in instructional quality. Our analysis highlights GPT-4o’s limitations in weak reflective questioning, overuse of generic praise, a condescending tone, and a tendency to overwhelm novices with excessive suggestions.

💡 Research Summary

The paper introduces SimInstruct, a scalable expert‑in‑the‑loop tool for generating high‑quality, multi‑turn scaffolding dialogues between human experts and LLM‑simulated novices. Recognizing the scarcity of authentic instructional dialogue data—largely due to privacy concerns and the vulnerability inherent in help‑seeking situations—the authors propose a synthetic yet pedagogically rich alternative. Using teacher‑development coaching as a case study, SimInstruct creates diverse novice personas by randomly combining domain attributes (e.g., classroom context, teaching experience, discipline) with four Big‑Five personality traits (extroversion, openness, conscientiousness, agreeableness). A verification step using GPT‑4 ensures logical consistency of each persona.

The workflow proceeds in three stages. First, a persona generator produces a detailed profile for each simulated novice. Second, GPT‑4 crafts an initial, context‑specific question that the novice poses to the expert. Third, a “turbo‑preview” LLM generates follow‑up novice responses that stay faithful to the persona’s style and traits. Human experts then engage in natural, multi‑turn conversations, typically following a three‑phase scaffolding structure: problem identification, reason exploration, and strategy development.

Data collection involved 18 experienced teacher‑development experts in the United States, who asynchronously interacted with the system over a two‑week period. The study yielded 123 dialogues, comprising 1,848 turns, an average of 15 turns per dialogue, 528 words contributed by the simulated novices, and 313 words by the experts. Statistical analysis revealed that novices with high extroversion prompted significantly longer expert contributions (≈ 87 additional words per dialogue), whereas agreeableness, conscientiousness, and openness showed no reliable effects. Discipline‑specific variations were also observed, with fields such as Earth Science and Nursing generating longer exchanges.

To assess realism, the authors compared four SimInstruct dialogues with four real, face‑to‑face coaching sessions (totaling ~200 minutes). Despite the limited sample, the synthetic dialogues fell within the same turn‑range distribution and were judged comparable in pedagogical relevance, cognitive depth, and metacognitive prompting. Expert participants reported that interacting with simulated novices was engaging, reflective, and even reinforced their own instructional judgment—a benefit aligned with the “learning‑by‑teaching” paradigm.

The collected corpus was then used to fine‑tune a LLaMA‑based expert model. In head‑to‑head evaluations, this fine‑tuned model outperformed GPT‑4o on several instructional quality metrics: it asked deeper reflective questions, provided more concrete feedback, avoided over‑generic praise, and maintained a supportive rather than condescending tone. The authors highlight GPT‑4o’s shortcomings—weak reflective questioning, excessive generic praise, occasional patronizing language, and a tendency to overwhelm novices with too many suggestions.

Key contributions of the work are: (1) a responsible, privacy‑preserving pipeline for large‑scale scaffolding dialogue generation; (2) empirical evidence that persona design (especially extroversion) materially influences dialogue richness; (3) demonstration that synthetic, expert‑curated data can improve the pedagogical performance of LLMs beyond state‑of‑the‑art models; and (4) insight that expert participants gain professional reflection through the data‑collection process itself.

Future directions include extending the approach to other professional domains (e.g., clinical supervision, counseling), enriching persona attributes with cultural or demographic factors, developing automated quality‑assessment metrics, and integrating real‑time AI assistance for live coaching scenarios. Overall, SimInstruct offers a promising avenue for bridging the data gap in AI‑supported education while simultaneously fostering expert development.