Watch and Listen: Understanding Audio-Visual-Speech Moments with Multimodal LLM

Humans naturally understand moments in a video by integrating visual and auditory cues. For example, localizing a scene in the video like “A scientist passionately speaks on wildlife conservation as dramatic orchestral music plays, with the audience nodding and applauding” requires simultaneous processing of visual, audio, and speech signals. However, existing models often struggle to effectively fuse and interpret audio information, limiting their capacity for comprehensive video temporal understanding. To address this, we present TriSense, a triple-modality large language model designed for holistic video temporal understanding through the integration of visual, audio, and speech modalities. Central to TriSense is a Query-Based Connector that adaptively reweights modality contributions based on the input query, enabling robust performance under modality dropout and allowing flexible combinations of available inputs. To support TriSense’s multimodal capabilities, we introduce TriSense-2M, a high-quality dataset of over 2 million curated samples generated via an automated pipeline powered by fine-tuned LLMs. TriSense-2M includes long-form videos and diverse modality combinations, facilitating broad generalization. Extensive experiments across multiple benchmarks demonstrate the effectiveness of TriSense and its potential to advance multimodal video analysis. Code and dataset will be publicly released.

💡 Research Summary

The paper introduces TriSense, a triple‑modality large language model (LLM) that jointly processes visual, audio, and speech streams to achieve fine‑grained temporal understanding of videos. The authors identify two major bottlenecks in current multimodal video models: (1) the scarcity of large‑scale, high‑quality datasets that contain synchronized annotations for all three modalities, especially in long‑form videos, and (2) the lack of mechanisms that can dynamically assess and re‑weight the relevance of each modality according to the user’s query.

To address (1), they construct TriSense‑2M, a curated dataset of over 2 million video samples. Each sample includes three separate captions: a visual caption describing observable actions and objects, an audio caption describing ambient sounds and music, and a speech caption that is a transcript of spoken language. The videos are long (average length 905 seconds, with many exceeding 10 minutes), which is far longer than most existing benchmarks. The dataset is built through an automated pipeline: first, raw videos are taken from InternVid and VAST, then modality‑specific captions are generated using existing pipelines. Two custom LLMs— a Generator and a Judger— are fine‑tuned on a high‑quality reference corpus (derived from GPT‑4‑o1 and manually refined). The Generator fuses the three unimodal captions into three multimodal forms (Audio‑Visual‑Speech, Audio‑Visual, Visual‑Speech). The Judger scores each fused caption on a 0‑5 semantic alignment scale against the original unimodal inputs; only captions scoring ≥ 3 are retained. This filtering yields a high‑quality, diverse set that explicitly includes cases where one or more modalities are missing, thereby supporting “modality dropout” training.

For (2), the core architectural contribution is the Query‑Based Connector. When a natural‑language query is received, the connector parses its semantics and computes adaptive weights for the visual, audio, and speech token streams. These weights modulate the multi‑head attention layers so that the most relevant modality (as inferred from the query) dominates the representation, while irrelevant or absent modalities are down‑weighted. This dynamic re‑weighting enables robust performance even when some modalities are noisy, partially corrupted, or completely absent—a scenario where most prior multimodal LLMs fail.

TriSense is evaluated on two fundamental video‑temporal tasks: segment captioning (generating a description for a given time interval) and moment retrieval (localizing the temporal span that matches a textual query). Experiments cover eight modality configurations: all three modalities, any two‑modality pair, and each single modality alone. Both zero‑shot (no task‑specific fine‑tuning) and fine‑tuned settings are tested on TriSense‑2M and on public benchmarks such as Charades‑ST, ActivityNet‑Captions, VAST‑27M, and LongV ALE. Results show that TriSense consistently outperforms state‑of‑the‑art models (e.g., LongV ALE, Qwen2.5‑Omni, VTG‑LLM) by 4–7 absolute points on BLEU‑4, METEOR, and Recall@1 when all modalities are present. More strikingly, under modality‑dropout conditions (e.g., audio missing, speech corrupted), TriSense gains 12–18 % relative improvement over baselines, confirming the effectiveness of the Query‑Based Connector. Ablation studies reveal that removing the connector degrades performance dramatically, and that the Judger‑based filtering step contributes a measurable boost to overall accuracy.

The authors acknowledge limitations: the current pipeline still relies on an external speech‑to‑text front‑end, and extremely complex acoustic scenes with overlapping speakers and music can challenge the weight estimation. Moreover, training the Generator and Judger required a substantial amount of high‑quality GPT‑4‑derived data, which may be costly for other groups to replicate.

Future work is outlined along three directions: (i) integrating end‑to‑end speech recognition to eliminate the preprocessing bottleneck, (ii) designing more sophisticated temporal synchronization mechanisms (e.g., time‑aware cross‑modal attention) to better handle long‑range dependencies, and (iii) extending the model to real‑time streaming scenarios where modalities arrive asynchronously.

In summary, TriSense advances multimodal video understanding by (1) providing a large, long‑form, multimodal dataset that explicitly models modality dropout, and (2) introducing a query‑driven adaptive fusion mechanism that lets a single LLM flexibly attend to visual, audio, and speech cues. This combination yields state‑of‑the‑art performance on both captioning and retrieval tasks and opens new possibilities for applications such as automated video summarization, multimodal search engines, and interactive AI assistants that can truly “watch and listen.”