SkyReels-V3 Technique Report

Video generation serves as a cornerstone for building world models, where multimodal contextual inference stands as the defining test of capability. In this end, we present SkyReels-V3, a conditional video generation model, built upon a unified multimodal in-context learning framework with diffusion Transformers. SkyReels-V3 model supports three core generative paradigms within a single architecture: reference images-to-video synthesis, video-to-video extension and audio-guided video generation. (i) reference images-to-video model is designed to produce high-fidelity videos with strong subject identity preservation, temporal coherence, and narrative consistency. To enhance reference adherence and compositional stability, we design a comprehensive data processing pipeline that leverages cross frame pairing, image editing, and semantic rewriting, effectively mitigating copy paste artifacts. During training, an image video hybrid strategy combined with multi-resolution joint optimization is employed to improve generalization and robustness across diverse scenarios. (ii) video extension model integrates spatio-temporal consistency modeling with large-scale video understanding, enabling both seamless single-shot continuation and intelligent multi-shot switching with professional cinematographic patterns. (iii) Talking avatar model supports minute-level audio-conditioned video generation by training first-and-last frame insertion patterns and reconstructing key-frame inference paradigms. On the basis of ensuring visual quality, synchronization of audio and videos has been optimized. Extensive evaluations demonstrate that SkyReels-V3 achieves state-of-the-art or near state-of-the-art performance on key metrics including visual quality, instruction following, and specific aspect metrics, approaching leading closed-source systems. Github: https://github.com/SkyworkAI/SkyReels-V3.

💡 Research Summary

SkyReels‑V3 is a conditional video generation framework that unifies three major generative paradigms—reference image‑to‑video synthesis, video extension, and audio‑guided talking‑avatar generation—within a single diffusion‑Transformer architecture. The authors emphasize a multimodal in‑context learning paradigm that accepts visual references (up to four images), an input video clip, an audio waveform, and a textual prompt, allowing flexible and controllable video synthesis across a wide range of scenarios.

Reference Image‑to‑Video
To mitigate copy‑paste artifacts common in earlier image‑to‑video pipelines, the authors construct a dedicated data processing pipeline. High‑quality video clips are first filtered from a massive in‑house dataset. Cross‑frame pairing selects temporally diverse frames, after which image‑editing models extract subject masks and perform background completion. Semantic rewriting further refines the pairs, reducing spurious artifacts. During training, an image‑video hybrid strategy jointly optimizes on static image datasets and dynamic video datasets, while multi‑resolution joint optimization enables the model to handle various output aspect ratios (1:1, 3:4, 4:3, 16:9, 9:16) and resolutions up to 720p. The conditioning mechanism concatenates VAE‑encoded latents of each reference image with the video latent, supporting up to four references and providing fine‑grained control over subject appearance and scene composition.

Video Extension
The extension module can continue a given clip (single‑shot extension) or perform shot‑switching with five predefined transition types (cut‑in, cut‑out, multi‑angle, shot/reverse‑shot, cut‑away). A shot‑switching detector analyses long‑form videos to identify transition boundaries and classify their type, which guides the construction of training data. A unified multi‑segment positional encoding combined with hierarchical training enables accurate motion modeling across segment boundaries. The system produces 720p outputs with adjustable durations from 5 to 30 seconds, preserving motion dynamics, visual style, and narrative coherence.

Talking Avatar
The avatar branch generates minute‑level videos from a single portrait image and an audio clip. It employs a first‑and‑last frame insertion pattern to anchor identity at the start and end of the generated sequence, allowing a single forward pass to synthesize up to one minute of coherent video. Phoneme‑level alignment losses enforce precise lip‑sync, while the model supports multiple visual styles (photorealistic, cartoon, animal) and multi‑character scenes (requiring explicit masks to indicate the speaking character).

Evaluation
A benchmark of 200 diverse reference‑image/video pairs (film, TV, e‑commerce, advertising) is used to assess three metrics: Reference Consistency (identity, clothing, object, background), Instruction Following (semantic adherence), and Visual Quality (image fidelity, motion smoothness, aesthetics). Table 1 shows SkyReels‑V3 achieving the highest Visual Quality (0.8119) and competitive Reference Consistency (0.6698) compared to leading commercial and open‑source models such as Kling, PixVerse V5, and VEO. Qualitative examples (Figures 1‑10) demonstrate robust multi‑subject interactions, seamless shot transitions, and high‑fidelity audio‑visual synchronization.

Significance and Limitations
SkyReels‑V3 showcases that a unified multimodal diffusion model can handle complex video generation tasks without task‑specific fine‑tuning. The combination of a sophisticated data pipeline, hybrid image‑video training, and multi‑resolution optimization is identified as the key driver of its performance. Limitations include a maximum output resolution of 720p, lack of real‑time inference optimization, and the need for manual mask specification in multi‑character avatar scenarios.

Conclusion
By integrating image‑to‑video synthesis, video continuation, and audio‑conditioned avatar generation into a single, open‑source framework, SkyReels‑V3 pushes the state of the art in multimodal video generation and provides a solid foundation for future research on higher resolutions, real‑time applications, and broader domain adaptation.