A Cocktail-Party Benchmark: Multi-Modal dataset and Comparative Evaluation Results

We introduce the task of Multi-Modal Context-Aware Recognition (MCoRec) in the ninth CHiME Challenge, which addresses the cocktail-party problem of overlapping conversations in a single-room setting using audio, visual, and contextual cues. MCoRec captures natural multi-party conversations where the recordings focus on unscripted, casual group chats, leading to extreme speech overlap of up to 100% and highly fragmented conversational turns. The task requires systems to answer the question “Who speaks when, what, and with whom?” by jointly transcribing each speaker’s speech and clustering them into their respective conversations from audio-visual recordings. Audio-only baselines exceed 100% word error rate, whereas incorporating visual cues yields substantial 50% improvements, highlighting the importance of multi-modality. In this manuscript, we present the motivation behind the task, outline the data collection process, and report the baseline systems developed for the MCoRec.

💡 Research Summary

The paper introduces the Multi‑Modal Context‑Aware Recognition (MCoRec) task as part of the ninth CHiME Challenge, targeting the classic “cocktail‑party” problem where multiple conversations overlap in a single room. To support this task, the authors release a novel dataset that captures up to eight participants engaging in up to four simultaneous conversations. Recordings consist of a central 360° video (GoPro Max) with its built‑in microphone, plus a personal smartphone video and lavalier microphone for each participant. Ten different indoor environments (living rooms, meeting rooms, lecture halls, etc.) were used, providing diverse acoustic and visual conditions. Each session lasts about six minutes, and the dataset contains 150 sessions split into training (56 sessions, 5.6 h), development (25 sessions, 2.5 h) and evaluation (69 sessions, 6.9 h). The training split also includes the personal videos, enabling data‑augmentation strategies; the dev and test splits are limited to the central 360° view to reflect realistic deployment scenarios.

The annotation pipeline involves precise temporal alignment using a whistle cue, manual transcription of utterances (max 15 s each) from the high‑quality personal audio, face tracking on the stitched 360° video to obtain bounding‑box trajectories for each speaker, and assignment of group IDs that define which speakers belong to the same conversation. This results in a richly labeled multimodal corpus with synchronized audio‑visual streams, speaker bounding boxes, and conversation‑group labels.

MCoRec is formally defined as a function f(V, {Bi}) → ({Ŷi}, Ĉ) where V is the 360° video, Bi are the bounding‑box sequences for each target speaker si, Ŷi is the transcription for si, and Ĉ assigns speakers to conversation clusters. Evaluation combines three metrics: (1) speaker‑wise Word Error Rate (WER), (2) conversation‑clustering pairwise F1 score, and (3) a Joint Error = 0.5 · WER + 0.5 · (1 – F1). The joint metric balances transcription accuracy and clustering quality, ranging from 0 (perfect) to 1 (worst).

The baseline system follows a cascade architecture:

1. Active Speaker Detection (ASD) – a lightweight CNN‑GRU model that fuses audio and visual features to predict frame‑level speaking activity. On the dev set it achieves a micro‑averaged Intersection‑over‑Union of 75.58 %, indicating reasonable but improvable detection of active speech segments.

2. Audio‑Visual Speech Recognition (AVSR) – several state‑of‑the‑art models are evaluated: AV‑HuBERT CTC/Attention, AutoAVSR (Conformer encoder), Muavic‑EN (AV‑HuBERT encoder + Transformer decoder), and Llama‑AVSR (AV‑HuBERT + Whisper‑medium fed into Llama‑3.1‑8B). The off‑the‑shelf AV‑HuBERT CTC/Attention model performs best with 55.36 % WER on the dev set. By augmenting the training data (pairing each 360° segment with its corresponding personal view) and fine‑tuning AV‑HuBERT on ~104 h of audio‑visual material, the WER drops to 49.90 % (≈10 % relative improvement). Error analysis shows insertion errors dominate, reflecting over‑generation in highly overlapping speech.

3. Conversation Clustering – a time‑based approach that computes a pairwise score for each speaker pair: Score(i,j) = 1 – (overlap duration / total duration). Overlap is treated as evidence of different conversations, while non‑overlapping sequential speech suggests the same conversation. Scores are converted to distances, and agglomerative clustering with a 0.3 distance threshold groups speakers. This yields a conversation‑clustering F1 of 0.8153 on the dev set.

Combining the three modules, the overall Joint ASR‑Clustering Error Rate on the development set is 0.3548, indicating substantial room for improvement. The authors emphasize that audio‑only baselines exceed 100 % WER, while incorporating visual cues halves the error, underscoring the critical role of multimodality.

In conclusion, the paper delivers a comprehensive benchmark for multimodal, context‑aware speech recognition in realistic cocktail‑party environments. By providing the dataset, detailed annotation procedures, clear evaluation metrics, and a reproducible baseline pipeline, it establishes a solid foundation for future research. Potential extensions include leveraging gaze and gesture cues, graph‑based modeling of conversational dynamics, more sophisticated speaker diarization, and end‑to‑end architectures that jointly optimize transcription and clustering. The dataset and baseline code are publicly released, and submitted systems will be presented at the CHiME‑9 workshop alongside ICASSP 2026.