ALIVE: Animate Your World with Lifelike Audio-Video Generation

Video generation is rapidly evolving towards unified audio-video generation. In this paper, we present ALIVE, a generation model that adapts a pretrained Text-to-Video (T2V) model to Sora-style audio-video generation and animation. In particular, the model unlocks the Text-to-Video&Audio (T2VA) and Reference-to-Video&Audio (animation) capabilities compared to the T2V foundation models. To support the audio-visual synchronization and reference animation, we augment the popular MMDiT architecture with a joint audio-video branch which includes TA-CrossAttn for temporally-aligned cross-modal fusion and UniTemp-RoPE for precise audio-visual alignment. Meanwhile, a comprehensive data pipeline consisting of audio-video captioning, quality control, etc., is carefully designed to collect high-quality finetuning data. Additionally, we introduce a new benchmark to perform a comprehensive model test and comparison. After continue pretraining and finetuning on million-level high-quality data, ALIVE demonstrates outstanding performance, consistently outperforming open-source models and matching or surpassing state-of-the-art commercial solutions. With detailed recipes and benchmarks, we hope ALIVE helps the community develop audio-video generation models more efficiently. Official page: https://github.com/FoundationVision/Alive.

💡 Research Summary

ALIVE (Animate Your World with Lifelike Audio‑Video Generation) is a unified audio‑video generation framework that extends a pretrained text‑to‑video (T2V) foundation model to support simultaneous high‑fidelity video synthesis, synchronized speech, background sounds, and reference‑based animation. The authors build on the Waver 1.0 video diffusion transformer and introduce a “Dual Stream + Single Stream” architecture: a Video‑DiT processes visual latents while a newly designed Audio‑DiT processes compressed audio latents obtained via a Wave‑VAE encoder.

The core technical contribution lies in the temporally‑aligned cross‑modal interaction. Standard cross‑attention ignores physical time, which leads to poor lip‑sync especially in multi‑speaker scenarios. ALIVE solves this with two novel components: TA‑CrossAttn (Temporally‑Aligned Cross‑Attention) and UniTemp‑RoPE (Unified Temporal Rotary Positional Embedding). UniTemp‑RoPE maps video latent positions to continuous timestamps in the audio coordinate system, turning discrete token indices into real‑valued temporal positions that reflect the actual sampling rates of audio (e.g., 16 kHz) and video (e.g., 30 fps). TA‑CrossAttn then incorporates these continuous positions into the attention mechanism, and a timestep‑aware adaptive layer‑norm normalizes the cross‑modal signals before fusion, preventing one modality from dominating the other. In early (dual‑stream) layers the interaction is bidirectional, allowing video to guide audio onset timing and audio to guide visual texture; in later (single‑stream) layers it becomes unidirectional (audio → video) to preserve the already‑established audio structure while refining visual details.

Audio generation is conditioned on two separate textual streams. Speech transcripts are wrapped in a tag and fed directly to the Audio‑DiT, ensuring that the exact phonemic content is reproduced. A global caption, enriched with tags for sound events and natural‑language description, is encoded by the large language model Qwen‑2.5‑32B‑Instruct and injected via cross‑attention to control timbre, emotion, and background ambience. This dual‑conditioning yields intelligible speech with controllable style while preserving the ability to add non‑speech sounds (e.g., door opening).