LipSody: Lip-to-Speech Synthesis with Enhanced Prosody Consistency

Lip-to-speech synthesis aims to generate speech audio directly from silent facial video by reconstructing linguistic content from lip movements, providing valuable applications in situations where audio signals are unavailable or degraded. While recent diffusion-based models such as LipVoicer have demonstrated impressive performance in reconstructing linguistic content, they often lack prosodic consistency. In this work, we propose LipSody, a lip-to-speech framework enhanced for prosody consistency. LipSody introduces a prosody-guiding strategy that leverages three complementary cues: speaker identity extracted from facial images, linguistic content derived from lip movements, and emotional context inferred from face video. Experimental results demonstrate that LipSody substantially improves prosody-related metrics, including global and local pitch deviations, energy consistency, and speaker similarity, compared to prior approaches.

💡 Research Summary

LipSody is a diffusion‑based lip‑to‑speech framework that explicitly targets prosody consistency while preserving the high intelligibility of state‑of‑the‑art models such as LipVoicer. The authors observe that existing lip‑to‑speech systems, despite achieving impressive word error rates (WER), often generate speech with flat or mismatched pitch and energy contours, resulting in unnatural and speaker‑inconsistent output. To address this, LipSody introduces a “prosody‑guiding” strategy that leverages three complementary visual cues: (1) speaker identity extracted from a full‑face image, (2) linguistic content derived from a lip‑centered video sequence, and (3) emotional context inferred from the entire face video using an EmoCLIP‑based encoder.

The core architecture builds on LipVoicer’s conditional denoising diffusion probabilistic model (DDPM) with classifier‑free guidance (CFG). During training, ground‑truth pitch (p) and energy (e) are extracted from paired audio and injected as oracle supervision. A speaker‑wise waveform normalization is applied instead of the usual clip‑wise normalization, preserving each speaker’s characteristic energy distribution. At inference time, a separate prosody prediction network estimates frame‑wise pitch and energy from the concatenated embeddings (speaker, content, emotion). These predicted prosodic features ( (\hat p), (\hat e) ) are fed back into the diffusion step, modifying the noise prediction as:

\