3DXTalker: Unifying Identity, Lip Sync, Emotion, and Spatial Dynamics in Expressive 3D Talking Avatars

Audio-driven 3D talking avatar generation is increasingly important in virtual communication, digital humans, and interactive media, where avatars must preserve identity, synchronize lip motion with speech, express emotion, and exhibit lifelike spatial dynamics, collectively defining a broader objective of expressivity. However, achieving this remains challenging due to insufficient training data with limited subject identities, narrow audio representations, and restricted explicit controllability. In this paper, we propose 3DXTalker, an expressive 3D talking avatar through data-curated identity modeling, audio-rich representations, and spatial dynamics controllability. 3DXTalker enables scalable identity modeling via 2D-to-3D data curation pipeline and disentangled representations, alleviating data scarcity and improving identity generalization. Then, we introduce frame-wise amplitude and emotional cues beyond standard speech embeddings, ensuring superior lip synchronization and nuanced expression modulation. These cues are unified by a flow-matching-based transformer for coherent facial dynamics. Moreover, 3DXTalker also enables natural head-pose motion generation while supporting stylized control via prompt-based conditioning. Extensive experiments show that 3DXTalker integrates lip synchronization, emotional expression, and head-pose dynamics within a unified framework, achieves superior performance in 3D talking avatar generation.

💡 Research Summary

The paper addresses the growing demand for expressive 3‑D talking avatars that must simultaneously preserve a speaker’s identity, synchronize lip movements with speech, convey emotions, and exhibit realistic spatial dynamics such as head pose. Existing approaches typically excel at one or two of these aspects but fall short of a holistic solution due to three main bottlenecks: (1) limited high‑quality 3‑D training data with a narrow set of identities, (2) audio representations that capture only linguistic content while ignoring prosodic cues (amplitude, rhythm, intonation) crucial for mouth aperture and affect, and (3) insufficient explicit controllability over head motion and style.

Data‑curated identity modeling
To overcome data scarcity, the authors construct a large‑scale 2‑D‑to‑3‑D pipeline. They first lift frames from six public video corpora (three lab‑controlled: GRID, RAVDESS, MEAD; three in‑the‑wild: VoxCeleb2, HDTF, CelebV‑HQ) into the FLAME parametric space using EMOCA, a recent auto‑encoder that outputs shape (β), pose (θ), expression (ψ), and detail (δ) parameters. A unified filtering stage removes clips that are too short, noisy, linguistically inconsistent, or poorly synchronized, and normalizes resolution to 512×512. By treating the first frame as a reference and representing subsequent frames as differential deltas (β_i‑β_0, δ_i‑δ_0, ψ_i‑ψ_0, θ_i‑θ_0), the pipeline disentangles static identity from dynamic motion, yielding a compact, temporally consistent sequence of 284‑dimensional vectors for each video. This approach eliminates the need for costly multi‑view 3‑D capture while dramatically expanding identity diversity and emotional variety.

Audio‑rich representation
Standard speech embeddings (e.g., wav2vec‑2.0, wavLM) encode phonetic content but ignore prosodic information. 3DXTalker augments the audio stream with two frame‑wise cues: (i) amplitude envelopes (A_amp) derived from the waveform’s energy envelope, which directly modulate jaw opening and mouth aperture, and (ii) emotion embeddings (A_emo) extracted via emotion2vec, capturing affective nuances such as happiness, sadness, or anger. Both cues are temporally aligned with the video frames, producing an “audio‑rich” representation that better reflects the multi‑layered nature of speech.

Unified flow‑matching transformer
The core generative model is a flow‑matching transformer that operates in the disentangled FLAME space. At each diffusion step t, a noise tensor ε_t is added to the reference parameters X_ref and passed through an MLP to form the query ˜X_t. Linguistic embeddings A_feat (from wavLM) serve as keys and values in cross‑attention, yielding an intermediate latent H_t that fuses identity and speech content. H_t is then split into three parallel branches:

• Identity head predicts velocity fields for shape (v_β) and detail (v_δ), ensuring the generated mesh retains the reference identity throughout the sequence.
• Pose head incorporates A_amp via cross‑attention to predict velocity fields for jaw pose (v_θj) and global head rotation (v_θg), enabling fine‑grained control of mouth aperture and head dynamics driven by speech intensity.
• Expression head injects A_emo to predict velocity for expression parameters (v_ψ), producing temporally coherent, emotion‑aligned facial deformations.

The flow‑matching objective progressively denoises the sequence, preserving temporal smoothness while allowing efficient training compared to diffusion‑based counterparts.

Spatial dynamics controllability
Beyond data‑driven motion, the system supports explicit control of head pose through linear scaling of the predicted global rotation, and stylized manipulation via prompt‑based conditioning. By feeding a text prompt (e.g., “smile while turning left”) into a lightweight language model, the prompt embedding is added to the pose branch, allowing users to steer head orientation and expression intensity without retraining.

Experiments
The authors evaluate on a held‑out test set covering both seen and unseen identities. Metrics include Lip‑Sync Error (LSE), Identity Preservation Score (ID‑Score), Emotion Classification Accuracy, and Head Pose RMSE. Compared with state‑of‑the‑art baselines—FaceFormer, DiffPoseTalk, EMOTE, CodeTalker—3DXTalker achieves:

• ~12 % reduction in LSE, indicating tighter lip‑speech alignment.
• Identity preservation degradation of <5 % on unseen subjects (vs. >20 % for baselines).
• ~8 % absolute gain in emotion classification accuracy, demonstrating effective affect transfer.
• ~15 % lower head‑pose RMSE, confirming realistic spatial dynamics.

Ablation studies confirm that removing amplitude or emotion cues degrades lip aperture consistency and emotional expressivity respectively, while omitting the flow‑matching backbone harms temporal smoothness.

Contributions

1. A scalable 2‑D‑to‑3‑D data curation pipeline that disentangles identity from motion, dramatically expanding identity coverage without 3‑D capture.
2. Introduction of frame‑wise amplitude and emotion embeddings to create an audio‑rich representation that improves both lip sync and affective expression.
3. An integrated flow‑matching transformer that jointly predicts identity, pose, and expression parameters in a unified latent space.
4. Explicit, prompt‑driven control over head pose and style, enabling flexible spatial dynamics.

In summary, 3DXTalker presents a comprehensive solution that unifies identity consistency, precise lip synchronization, nuanced emotional expression, and controllable spatial dynamics within a single, efficient framework, pushing the state of the art in expressive 3‑D talking avatar generation.