Unlocking Large Audio-Language Models for Interactive Language Learning

Achieving pronunciation proficiency in a second language (L2) remains a challenge, despite the development of Computer-Assisted Pronunciation Training (CAPT) systems. Traditional CAPT systems often provide unintuitive feedback that lacks actionable guidance, limiting its effectiveness. Recent advancements in audio-language models (ALMs) offer the potential to enhance these systems by providing more user-friendly feedback. In this work, we investigate ALMs for chat-based pronunciation training by introducing L2-Arctic-plus, an English dataset with detailed error explanations and actionable suggestions for improvement. We benchmark cascaded ASR+LLMs and existing ALMs on this dataset, specifically in detecting mispronunciation and generating actionable feedback. To improve the performance, we further propose to instruction-tune ALMs on L2-Arctic-plus. Experimental results demonstrate that our instruction-tuned models significantly outperform existing baselines on mispronunciation detection and suggestion generation in terms of both objective and human evaluation, highlighting the value of the proposed dataset.

💡 Research Summary

The paper addresses the long‑standing challenge of providing effective pronunciation feedback to second‑language (L2) learners. Traditional computer‑assisted pronunciation training (CAPT) systems typically deliver phoneme‑ or word‑level error locations and overall scores, but the feedback is often abstract and difficult for learners to translate into concrete corrective actions. Leveraging recent advances in large audio‑language models (ALMs), the authors propose a chat‑based pronunciation training framework that generates natural‑language explanations of mispronunciations together with actionable suggestions (e.g., practice with similar‑sounding words, avoid vocal fold vibration).

To enable systematic evaluation, the authors extend the publicly available L2‑Arctic corpus, creating L2‑Arctic‑plus. The original dataset contains 900 utterances from 24 non‑native English speakers, annotated with binary mispronunciation flags and error types (substitution, deletion, insertion) at the phoneme level. Using these annotations, the authors first prompt GPT‑4o to produce structured error‑explanation and suggestion pairs for each mispronounced word. A two‑stage human verification process (three annotators) refines the automatically generated outputs, ensuring that each pair accurately reflects the underlying phonetic error and provides a realistic corrective strategy. The resulting dataset supplies both the canonical text and a rich, text‑based ground‑truth response (Y_R) for every sample, making it suitable for training and evaluating models that must output natural‑language feedback.

The study then evaluates two families of approaches on this benchmark. The first is a cascaded ASR + LLM pipeline: an automatic speech recognizer (Whisper Small/Medium/Large or wav2vec2 Base/Large) transcribes the audio, and a large language model (Mistral‑7B or Llama‑3.1‑8B) receives the canonical text, the ASR transcript, and a one‑shot demonstration to detect mismatches and generate feedback. Results show that stronger ASR models, which tend to “correct” mispronunciations during transcription, actually reduce the downstream LLM’s ability to spot errors, leading to low precision/recall and a high Extra Words Ratio (EWR) where the system hallucinations words not present in the original utterance.

The second family consists of off‑the‑shelf ALMs (e.g., Qwen‑Audio, SpeechGPT, Audio‑GPT, GPT‑4o). Despite their impressive capabilities on general speech and audio tasks, these models also struggle with the specific demands of mispronunciation detection and suggestion generation, yielding modest BLEU‑2, ROUGE‑L, and BERTScore values and frequently missing errors or inventing spurious words.

Recognizing these limitations, the authors fine‑tune ALMs directly on L2‑Arctic‑plus using an instruction‑tuning paradigm. The tuning pipeline freezes the audio encoder and projection layers while updating the language model’s parameters to map audio embeddings to the desired error‑explanation/suggestion pairs. Two-stage training (audio encoder → projector → LLM) is employed, and the models are trained on the curated pairs from L2‑Arctic‑plus. After instruction‑tuning, the ALM achieves a substantial boost in mispronunciation detection (F1 improves by roughly 20‑30 % over the best ASR + LLM baseline) and dramatically lowers EWR. In the feedback generation task, BLEU‑2 rises from ~4‑5 % to over 15 %, ROUGE‑L from ~6 % to 15 %, and BERTScore from ~6 % to nearly 12 %, indicating much higher lexical and semantic overlap with the human‑crafted references. Human evaluations corroborate these gains: annotators rate the tuned model’s suggestions as more relevant, clearer, and more helpful for actual pronunciation practice.

The paper’s contributions are threefold: (1) the release of L2‑Arctic‑plus, a novel benchmark that pairs audio with detailed textual feedback suitable for training chat‑style pronunciation assistants; (2) a systematic analysis showing that naïve ASR + LLM cascades and existing ALMs are insufficient for this task; (3) demonstration that instruction‑tuning an ALM on the new dataset yields state‑of‑the‑art performance in both detection and feedback generation. The work opens a pathway for deploying large multimodal models in language education, suggesting future directions such as multilingual extensions, real‑time interactive tutoring, and personalized feedback conditioned on learner profiles.