Autoregressive Video Autoencoder with Decoupled Temporal and Spatial Context

Video autoencoders compress videos into compact latent representations for efficient reconstruction, playing a vital role in enhancing the quality and efficiency of video generation. However, existing video autoencoders often entangle spatial and temporal information, limiting their ability to capture temporal consistency and leading to suboptimal performance. To address this, we propose Autoregressive Video Autoencoder (ARVAE), which compresses and reconstructs each frame conditioned on its predecessor in an autoregressive manner, allowing flexible processing of videos with arbitrary lengths. ARVAE introduces a temporal-spatial decoupled representation that combines downsampled flow field for temporal coherence with spatial relative compensation for newly emerged content, achieving high compression efficiency without information loss. Specifically, the encoder compresses the current and previous frames into the temporal motion and spatial supplement, while the decoder reconstructs the original frame from the latent representations given the preceding frame. A multi-stage training strategy is employed to progressively optimize the model. Extensive experiments demonstrate that ARVAE achieves superior reconstruction quality with extremely lightweight models and small-scale training data. Moreover, evaluations on video generation tasks highlight its strong potential for downstream applications.

💡 Research Summary

The paper introduces the Autoregressive Video Autoencoder (ARVAE), a novel framework that addresses the fundamental limitation of existing video autoencoders: the entanglement of spatial and temporal information. Traditional video AEs typically process fixed‑length clips with 3‑D convolutions or spatio‑temporal attention, mixing spatial details and motion cues and thereby failing to exploit the strong temporal redundancy between adjacent frames. ARVAE instead adopts a frame‑by‑frame autoregressive scheme: each frame is encoded and decoded conditioned on the previously reconstructed frame. This design mirrors the natural causality of video streams and enables flexible handling of videos of arbitrary length.

The core technical contribution is a temporal‑spatial decoupled latent representation. Temporal information is captured by a downsampled optical flow field (the “Temporal Motion”). A motion estimator based on SpyNet first predicts a high‑resolution flow M between the current frame Xₜ and the previous reconstructed frame \hat{X}{t‑1}. A Temporal Encoder then builds a multi‑scale pyramid of motion features and image features from \hat{X}{t‑1}, progressively down‑sampling them N times (compression ratio r = 2ᴺ). The smallest‑resolution motion feature becomes the compact temporal latent T, while the pyramid of warped image features constitutes “propagated features” Pₑ that carry structural information from the past frame.

Spatial information that cannot be transferred through motion—new objects, texture changes, illumination shifts—is encoded separately as a Spatial Supplement S. The Spatial Encoder receives Pₑ and the raw current frame Xₜ, concatenates them, and passes them through residual blocks at each scale to compute the residual (difference) between the warped prediction and the true frame. This residual is down‑sampled across scales, yielding a compact spatial latent S.

During decoding, the Temporal Decoder upsamples T back to full resolution, reconstructs the multi‑scale motion, and warps \hat{X}_{t‑1} to obtain a new set of propagated features P_d. The Spatial Decoder then upsamples S and merges it with P_d via residual blocks, producing the final reconstructed frame \hat{X}_t. The first frame of any sequence is handled by a standard image autoencoder (e.g., the FLUX VAE).

Training proceeds in multi‑stage stages. Early stages expose the model to short sequences (2–4 frames) and modest motion magnitudes, allowing stable learning of flow estimation and spatial residuals. Subsequent stages gradually increase sequence length and motion amplitude, encouraging the network to capture long‑range temporal dependencies. The loss combines reconstruction (L1/L2), flow regularization, and an entropy‑based compression term that penalizes the information content of T and S.

Empirical evaluation covers two dimensions. First, a Shannon entropy analysis on the MCL‑JCV dataset shows that the decoupled representation reduces entropy to roughly half of the raw frames, outperforming prior low‑/high‑frequency split methods. Second, reconstruction quality (PSNR, SSIM) is benchmarked against state‑of‑the‑art 3‑D‑Conv and attention‑based video AEs. Despite using 80× fewer parameters (≈10 M) and 6,700× less training data, ARVAE achieves comparable or superior PSNR (30‑35 dB) and SSIM, demonstrating that the autoregressive, decoupled design yields high compression efficiency without sacrificing fidelity.

The authors also test ARVAE as a latent space provider for downstream video generation. When paired with a latent video diffusion model, the temporally conditioned latents enable generation with excellent temporal coherence and fine‑grained texture detail, confirming the practical utility of the representation beyond reconstruction.

In summary, ARVAE contributes (1) an autoregressive encoding/decoding pipeline that naturally leverages inter‑frame redundancy, (2) a temporally downsampled flow latent plus a spatial residual latent that together preserve all visual information while drastically reducing bitrate, (3) a staged training regime that stabilizes learning of long‑term dependencies, and (4) strong empirical evidence that lightweight models trained on modest data can match or exceed the performance of much larger video autoencoders. This work therefore proposes a new paradigm for video compression and latent‑space modeling, with immediate implications for efficient video generation, editing, and streaming applications.