Multimodal Conversation Structure Understanding
While multimodal large language models (LLMs) excel at dialogue, whether they can adequately parse the structure of conversation – conversational roles and threading – remains underexplored. In this work, we introduce a suite of tasks and release TV-MMPC, a new annotated dataset, for multimodal conversation structure understanding. Our evaluation reveals that while all multimodal LLMs outperform our heuristic baseline, even the best-performing model we consider experiences a substantial drop in performance when character identities of the conversation are anonymized. Beyond evaluation, we carry out a sociolinguistic analysis of 350,842 utterances in TVQA. We find that while female characters initiate conversations at rates in proportion to their speaking time, they are 1.2 times more likely than men to be cast as an addressee or side-participant, and the presence of side-participants shifts the conversational register from personal to social.
💡 Research Summary
The paper tackles a largely unexplored problem in multimodal dialogue understanding: the ability of large language models (LLMs) to parse the structural aspects of conversation, namely who is speaking, who is being addressed, who is a side‑participant, and how utterances are linked in reply‑to threads. To this end, the authors introduce a new benchmark suite of four tightly coupled tasks—speaker identification, addressee detection, side‑participant detection, and conversation‑thread linking—and release TV‑MMPC, a human‑annotated dataset built on top of TVQA. TV‑MMPC contains 4,378 speaker annotations, 5,599 addressee annotations, and 3,412 side‑participant annotations across 200 randomly sampled 60‑90‑second clips from four popular TV series (The Big Bang Theory, Friends, House M.D., and How I Met Your Mother). The annotation pipeline combines automatic preprocessing (face detection, Whisper transcription, pyannote speaker diarization) with a dedicated web interface used by four authors, achieving inter‑annotator F1 scores in the mid‑80s, indicating high reliability.
The authors ground their task design in classic sociolinguistic theory (Goffman, Goodwin, Clark & Carlson), formalizing participant roles along three binary dimensions: addressed, ratified (i.e., recognized as part of the conversational group), and known (perceptually engaged). This yields a clear taxonomy where addressees are addressed, ratified, and known; side‑participants are ratified and known but not addressed; and bystanders are neither addressed nor ratified. Conversation disentanglement is defined as a reply‑to linking problem: each utterance either points to a preceding utterance it directly answers or points to itself to signal the start of a new thread.
For evaluation, the paper adopts a zero‑shot setting across six multimodal LLMs: four vision‑language models (LLaMA‑4‑Scout, GPT‑4.1‑mini, o4‑mini, Gemini 2.0‑Flash) that process sampled frames and subtitles, and two audio‑visual LLMs (Qwen 2.5‑Omni, Reka‑Flash) that ingest video, audio, and text. A simple heuristic baseline combines Whisper transcription, pyannote diarization, and face frequency heuristics to assign speakers, addressees (most frequent non‑speaker face in a short context window), side‑participants (remaining faces), and always links each utterance to its immediate predecessor. The baseline achieves modest scores (e.g., speaker F1 ≈ 34.7, addressee F1 ≈ 19.5, thread linking ≈ 86.7).
All multimodal LLMs outperform the baseline, with Gemini 2.0‑Flash achieving the best overall performance (speaker F1 ≈ 78.6, addressee F1 ≈ 78.6, side‑participant F1 ≈ 57.7, thread linking 1‑NVI ≈ 89.5%). Vision‑language models perform competitively on role attribution but lag behind audio‑visual models on thread linking, reflecting the benefit of audio cues for detecting reply relationships. However, a striking finding emerges when character names are anonymized (replaced with “Anonymous”). Across all models, speaker and addressee identification drop by roughly 10–15 percentage points, and side‑participant detection also suffers, indicating that current multimodal LLMs rely heavily on textual identity cues rather than visual or acoustic signals to resolve conversational roles.
Beyond model benchmarking, the authors conduct a large‑scale sociolinguistic analysis on the full TVQA corpus (350,842 utterances). They find that female characters initiate conversations in proportion to their overall speaking time, but they are 1.2 × more likely than male characters to be labeled as addressees or side‑participants. Moreover, the presence of side‑participants correlates with a shift in conversational register from “personal” (e.g., intimate, emotion‑laden exchanges) to “social” (e.g., information sharing, group coordination). These patterns suggest systematic gendered role allocation in scripted TV dialogue and demonstrate how multimodal structure analysis can surface cultural biases.
The paper discusses several limitations. First, the heavy dependence on character name information raises concerns about model robustness in anonymized or privacy‑sensitive settings (e.g., customer service bots). Second, the dataset is confined to scripted TV shows, which may not generalize to spontaneous, real‑world conversations such as meetings or online chats. Third, the annotation effort, while high‑quality, is limited in scale; scaling to larger corpora will likely require crowd‑sourcing or improved automatic labeling pipelines.
In conclusion, TV‑MMPC provides the first comprehensive, human‑annotated benchmark for multimodal conversation‑structure understanding, coupling role attribution with thread disentanglement. The experimental results reveal both the promise of current multimodal LLMs and their current shortcomings, especially regarding reliance on textual identity cues. Future work should aim to (1) develop models that can infer speaker and addressee purely from visual and acoustic cues, (2) expand the dataset to diverse domains and languages, and (3) integrate sociolinguistic bias detection into model training and evaluation pipelines. This work lays a solid foundation for building dialogue systems that understand not just what is said, but who is saying it, to whom, and how the conversation evolves over time.
Comments & Academic Discussion
Loading comments...
Leave a Comment