Beyond Lips: Integrating Gesture and Lip Cues for Robust Audio-visual Speaker Extraction

Most audio-visual speaker extraction methods rely on synchronized lip recording to isolate the speech of a target speaker from a multi-talker mixture. However, in natural human communication, co-speech gestures are also temporally aligned with speech, often emphasizing specific words or syllables. These gestures provide complementary visual cues that can be especially valuable when facial or lip regions are occluded or distant. In this work, we move beyond lip-centric approaches and propose SeLG, a model that integrates both lip and upper-body gesture information for robust speaker extraction. SeLG features a cross-attention-based fusion mechanism that enables each visual modality to query and selectively attend to relevant speech features in the mixture. To improve the alignment of gesture representations with speech dynamics, SeLG also employs a contrastive InfoNCE loss that encourages gesture embeddings to align more closely with corresponding lip embeddings, which are more strongly correlated with speech. Experimental results on the YGD dataset, containing TED talks, demonstrate that the proposed contrastive learning strategy significantly improves gesture-based speaker extraction, and that our proposed SeLG model, by effectively fusing lip and gesture cues with an attention mechanism and InfoNCE loss, achieves superior performance compared to baselines, across both complete and partial (i.e., missing-modality) conditions.

💡 Research Summary

The paper introduces SeLG (Speech extraction with Lip and co‑speech Gesture cues), a novel audio‑visual speaker extraction system that jointly leverages lip movements and upper‑body co‑speech gestures. Traditional audio‑visual speaker extraction (AVSE) methods rely almost exclusively on synchronized lip recordings; however, lip regions can be occluded, low‑resolution, or distant, limiting robustness. Human communication, by contrast, routinely pairs speech with co‑speech gestures that are temporally aligned with the spoken content and often emphasize key words or prosodic events. SeLG exploits this complementary visual modality to improve extraction performance, especially when one modality is missing or degraded.

Model Architecture
SeLG consists of five main components: a gesture encoder, a lip encoder, a speech encoder, a speech decoder, and a separator that performs cross‑attention‑based fusion.

• Gesture Encoder: Takes 3‑D coordinates of ten spine‑centered joints (head, neck, nose, spine, left/right shoulders, elbows, wrists) extracted from video via 2‑D pose detection followed by 3‑D lifting. These sequences are processed by a 5‑layer bidirectional LSTM (hidden size 32, dropout 0.3) to produce temporally aligned gesture embeddings V_g(t).
• Lip Encoder: Receives cropped lip images extracted by a face detector/tracker. It follows the visual encoder design of MuSE: a 3‑D convolutional front‑end, an 18‑layer residual network pre‑trained on visual speech recognition, and five stacked visual temporal convolutional blocks, yielding lip embeddings V_l(t).
• Speech Encoder/Decoder: A 1‑D convolution (in‑channel 1, out‑channel 256, kernel 40, stride 20) with ReLU converts the raw waveform x(τ) into frame‑wise latent representations X(t). The decoder reconstructs the enhanced waveform by element‑wise multiplication of X(t) with an estimated mask M̂(t) followed by overlap‑add.
• Separator: The core of SeLG. Two parallel transformer cross‑attention layers allow each visual modality to act as a query while the speech embedding X(t) supplies keys and values. The attention outputs from the lip and gesture streams are summed element‑wise, then fed into a dual‑path BLSTM (chunk size 100, hidden 128) that predicts the mask. This block is repeated R times (R = 5).

Contrastive Alignment (InfoNCE) Loss
Because gesture‑speech correlation is weaker than lip‑speech correlation, the authors introduce a contrastive loss to align gesture embeddings with their corresponding lip embeddings. For each time step i, V_g(i)·V_l(i) is treated as a positive pair, while V_g(i)·V_l(j) for j ≠ i are negatives. The loss is:

L_emb = – Σ_i log