Beyond Mimicry to Contextual Guidance: Knowledge Distillation for Interactive AI

As large language models increasingly mediate firm - customer interactions, firms face a tradeoff: the most capable models perform well but are costly and difficult to control at scale. Existing knowledge distillation methods address this challenge by training weaker, deployable models to imitate frontier outputs; however, such open-loop approaches are poorly suited to interactive, multi-turn settings where responses must be sequenced coherently across conversational states. We propose a shift in what knowledge is distilled - from output imitation to contextual guidance. We develop a framework in which a superior teacher model constructs a reusable library of strategic textual guidance for particular scenarios likely to be encountered by the student. When deployed, the student retrieves the context-specific guidance at inference time, enabling adaptive behavior without retraining. Using customer-service interactions, we show that this approach improves service quality and customer satisfaction relative to standard fine-tuning while maintaining alignment with firm policies. The results position inference-time textual guidance as a scalable and controllable approach to distillation for interactive AI agents in marketing settings.

💡 Research Summary

The paper tackles a pressing dilemma in enterprise AI: while the most capable large language models (LLMs) deliver superior performance in customer‑firm interactions, their high inference cost, privacy concerns, and limited controllability make them impractical for large‑scale deployment. Traditional knowledge distillation (KD) mitigates this gap by training a smaller “student” model to imitate the outputs of a powerful “teacher” model, but this open‑loop approach is ill‑suited for multi‑turn, goal‑oriented dialogs where early decisions shape the entire conversation trajectory.

To overcome these limitations, the authors propose a fundamentally different distillation paradigm: instead of transferring surface‑level responses, they transfer contextual strategic guidance. Their framework, named GER (Guidance Elicitation and Retrieval), consists of two tightly coupled phases.

Guidance elicitation (training phase). For each dialog scenario, the student first generates a response. The teacher then evaluates this response against its own expert behavior and produces a concise natural‑language instruction that specifies how the student should reason, what tactical move to make (e.g., probe, reassure, escalate), and which policy constraints to honor. This feedback—described as a “textual gradient”—is iteratively refined over multiple teacher‑student rounds until the guidance stabilizes. The resulting (state, guidance) pairs are stored in an external library, making the distilled knowledge explicit, inspectable, and editable.

Guidance retrieval (inference phase). When the deployed system encounters a new interaction, it computes a semantic embedding of the current conversational state, retrieves the most similar guidance from the library, and conditions the student’s next response on this guidance (e.g., by prepending it to the prompt). Because the guidance is external to the model weights, it can be updated instantly by human operators to reflect brand‑level policies, regulatory changes, or newly discovered best practices, without any retraining.

A critical engineering challenge is ensuring the library covers the state space the student will actually visit at deployment. Purely teacher‑driven data would leave large gaps, especially when the student deviates from the teacher’s trajectory, leading to out‑of‑distribution (OOD) states where no guidance exists. To address this, the authors adopt a closed‑loop scenario‑generation strategy inspired by interactive imitation learning: they begin with teacher‑only dialogs to harvest high‑quality guidance, then progressively incorporate student‑generated trajectories, expanding coverage to the student’s own error‑induced regions. This mitigates the classic behavioral cloning pitfall where a model fails to recover from its own mistakes.

Empirical evaluation. The authors test GER on a benchmark of multi‑turn, goal‑oriented customer‑service dialogs (based on the Bordes et al. 2017 dataset). They compare three families of baselines: (1) supervised fine‑tuning on teacher‑generated responses, (2) generic prompting without any distilled knowledge, and (3) parameter‑based KD variants that embed teacher behavior into weights. Evaluation combines automated metrics (BLEU, success rate, policy compliance) with a large‑scale human study measuring perceived reliability, empathy, responsiveness, and overall satisfaction.

Results show that GER consistently outperforms fine‑tuning and generic prompting across all metrics, often matching or exceeding the best parameter‑based KD models. Notably, the gains are most pronounced in dimensions tied to strategic decision‑making—students guided by GER are better at timing escalations, reframing objections, and maintaining brand‑consistent language. Human raters report higher satisfaction scores, attributing improvements to more coherent, empathetic, and policy‑aligned interactions. Moreover, inference‑time latency introduced by guidance retrieval is negligible (<30 ms), confirming suitability for real‑time deployment.

The authors also stress robustness: GER’s performance holds when (a) the student model is swapped for a different architecture (e.g., a 2.7 B vs. 6 B model), (b) synthetic training data replaces real dialogs, and (c) the system is transferred to related interaction domains (e.g., sales negotiations) without additional retraining—simply by curating a new guidance library.

Contributions.

1. Conceptual shift: redefining knowledge distillation from output imitation to transfer of contextual, strategic textual guidance.
2. GER framework: a concrete, repeatable pipeline for eliciting, refining, storing, and retrieving guidance, enabling closed‑loop, state‑dependent control at inference time.
3. Coverage‑driven scenario generation: a staged teacher‑student data collection process that expands guidance coverage to OOD states, ensuring robustness even with largely synthetic training pipelines.

In sum, the paper demonstrates that externalizing expert judgment as reusable natural‑language instructions can bridge the performance‑efficiency gap between frontier LLMs and deployable student models, delivering higher quality, policy‑compliant, and controllable interactive AI for marketing and customer‑service applications.