LPIPS-AttnWav2Lip: Generic Audio-Driven lip synchronization for Talking Head Generation in the Wild

Researchers have shown a growing interest in Audio-driven Talking Head Generation. The primary challenge in talking head generation is achieving audio-visual coherence between the lips and the audio, known as lip synchronization. This paper proposes a generic method, LPIPS-AttnWav2Lip, for reconstructing face images of any speaker based on audio. We used the U-Net architecture based on residual CBAM to better encode and fuse audio and visual modal information. Additionally, the semantic alignment module extends the receptive field of the generator network to obtain the spatial and channel information of the visual features efficiently; and match statistical information of visual features with audio latent vector to achieve the adjustment and injection of the audio content information to the visual information. To achieve exact lip synchronization and to generate realistic high-quality images, our approach adopts LPIPS Loss, which simulates human judgment of image quality and reduces instability possibility during the training process. The proposed method achieves outstanding performance in terms of lip synchronization accuracy and visual quality as demonstrated by subjective and objective evaluation results. The code for the paper is available at the following link: https://github.com/FelixChan9527/LPIPS-AttnWav2Lip

💡 Research Summary

**
The paper introduces LPIPS‑AttnWav2Lip, a generic audio‑driven talking‑head generation framework that achieves precise lip synchronization and high‑quality visual output for arbitrary speakers. Building upon the pioneering Wav2Lip and its attention‑enhanced variant AttnWav2Lip, the authors identify two critical shortcomings in existing methods: (1) the gradual attenuation of audio information as it passes through deep decoder layers, and (2) the simplistic concatenation of audio and visual embeddings, which fails to capture deep semantic relationships between speech content and mouth texture. To overcome these issues, the proposed architecture incorporates three major innovations.

First, a U‑Net generator is equipped with Residual Convolutional Block Attention Modules (CBAM). By sequentially applying channel‑wise and spatial attention within a residual block, the network emphasizes lip‑region features while preserving gradient flow, leading to more accurate reconstruction of the masked lower half of the face.

Second, a Semantic Alignment Module fuses audio and visual modalities more intelligently. It consists of a Fast Fourier Convolution (FFC) layer and Adaptive Instance Normalization (AdaIN). The FFC layer expands the receptive field through a dual‑branch design that captures both local convolutional details and global frequency‑domain context, essential for inpainting tasks where half of the face is masked. AdaIN then aligns the statistical moments (mean and variance) of visual feature maps with the audio latent vector, effectively injecting speech content into the visual domain without adding extra parameters.

Third, the training objective replaces the conventional adversarial loss with the Learned Perceptual Image Patch Similarity (LPIPS) loss. LPIPS measures perceptual similarity using deep feature activations from pretrained networks, encouraging the generator to produce images that are closer to human visual judgments rather than merely minimizing pixel‑wise errors. This substitution stabilizes GAN training, mitigates gradient vanishing/explosion, and yields superior perceptual quality.

The overall pipeline processes a masked lower‑face image together with five consecutive reference frames (providing temporal context) and the corresponding audio segment (converted to MFCCs). The face encoder (U‑Net encoder) extracts visual features, while a 2‑D CNN audio encoder produces a latent vector. These are merged via the Semantic Alignment Module and fed to the decoder, which reconstructs the full‑face frame. The loss function combines reconstruction loss, a synchronization loss based on a pre‑trained SyncNet (L_sync), and the LPIPS loss, with empirically set weights α=0.03 and β=0.07.

Extensive experiments on multi‑speaker datasets such as LRS2, LRS3, and VoxCeleb2 demonstrate that LPIPS‑AttnWav2Lip outperforms prior state‑of‑the‑art methods. Objective metrics show significant improvements in Lip‑Sync Error – Confidence (LSE‑C) and Lip‑Sync Error – Distance (LSE‑D), as well as a lower Fréchet Inception Distance (FID), indicating better visual fidelity. Subjective Mean Opinion Score (MOS) evaluations also favor the proposed method (4.3/5 vs. 3.7/5 for Wav2Lip). Ablation studies confirm the contribution of each component: removing CBAM degrades lip‑sync accuracy, omitting FFC harms global context modeling, and excluding AdaIN reduces the alignment between speech content and mouth shape.

The authors acknowledge remaining limitations, such as the reliance on a 2‑D CNN audio encoder that may struggle with very long utterances, and the focus on lower‑face inpainting which does not address full‑head or background dynamics. Future work includes integrating transformer‑based audio encoders for longer temporal dependencies, extending the framework to full‑body generation, and optimizing the model for real‑time deployment on edge devices.

In summary, LPIPS‑AttnWav2Lip presents a well‑balanced combination of attention‑enhanced feature extraction, sophisticated audio‑visual semantic alignment, and perceptual loss design, setting a new benchmark for speaker‑independent, high‑quality lip‑synchronised talking‑head synthesis.