Diffusion-aided Extreme Video Compression with Lightweight Semantics Guidance

Modern video codecs and learning-based approaches struggle for semantic reconstruction at extremely low bit-rates due to reliance on low-level spatiotemporal redundancies. Generative models, especially diffusion models, offer a new paradigm for video compression by leveraging high-level semantic understanding and powerful visual synthesis. This paper propose a video compression framework that integrates generative priors to drastically reduce bit-rate while maintaining reconstruction fidelity. Specifically, our method compresses high-level semantic representations of the video, then uses a conditional diffusion model to reconstruct frames from these semantics. To further improve compression, we characterize motion information with global camera trajectories and foreground segmentation: background motion is compactly represented by camera pose parameters while foreground dynamics by sparse segmentation masks. This allows for significantly boosts compression efficiency, enabling descent video reconstruction at extremely low bit-rates.

💡 Research Summary

The paper “Diffusion‑aided Extreme Video Compression with Lightweight Semantics Guidance” introduces a novel video compression framework that dramatically reduces bitrate while preserving semantic fidelity, by encoding high‑level motion semantics and using a conditional diffusion model for reconstruction. Traditional codecs (H.264/H.265/H.266) and recent learning‑based methods (e.g., DCVC‑FM) rely heavily on low‑level spatial and temporal redundancies; when pushed to extreme compression, they fail to retain meaningful content. To overcome this, the authors separate motion into two semantic streams: background motion is represented by camera pose trajectories (intrinsic matrix K and extrinsic parameters R, t), while foreground motion is captured by sparse segmentation masks generated via a hybrid pipeline that combines captioning, a large language model (LLM), and the Segment‑Anything Model 2 (SAM2).

For background, the method extracts per‑frame poses using the FlowMap optimizer, converts them to relative poses (E′i = E1⁻¹Ei), samples every other frame, and stores the differences (ΔE′) after quantization and Huffman coding. This reduces the background motion to roughly 12 N + 4 floating‑point numbers, a negligible overhead at low bitrates. For foreground, the caption model produces a short textual description of the scene; the LLM parses this description to identify moving objects, which are then fed as prompts to SAM2 to obtain consistent instance masks. These mask sequences are compressed with DCVC‑FM, which is well‑suited for binary or sparse maps.

The first intra‑frame of each Group‑of‑Pictures (GoP) is compressed using a diffusion‑based image codec, while all inter‑frames are reconstructed from the decoded semantics. At the decoder, missing camera poses are interpolated using spherical linear interpolation (Slerp) for rotations and linear interpolation for translations. The decoded pose sequence is transformed into Plücker embeddings (camera center and ray direction per pixel) and fused with noisy latent features inside a pretrained Stable Video Diffusion‑XL (SVD‑XL) model. Two lightweight adapters are trained: a pose adapter that injects the Plücker embeddings via addition and a linear layer before temporal attention, and a segmentation adapter that feeds the foreground masks through a ControlNet branch into the up‑sampling stage of the diffusion U‑Net. The adapters are concatenated and jointly fine‑tuned on RealEstate10K, then evaluated on DA‑VIS and MCL‑JCV datasets.

Experimental results show that the proposed CPSGD (Camera‑Pose and Segmentation Guided Diffusion) consistently outperforms H.264, H.265, DCVC‑FM, and a baseline diffusion codec across multiple perceptual metrics: LPIPS, FVD, CLIP‑Score, and subjective visual quality. At bit‑per‑pixel (BPP) values as low as 0.02, CPSGD achieves up to 50 % lower distortion than the best learning‑based baseline, while preserving object identity and motion semantics. A detailed bitrate breakdown reveals that textual descriptions occupy less than 2 % of the total stream, whereas camera pose and segmentation each contribute roughly 45 % of the bits, confirming that high‑level semantics dominate the compression budget.

The paper’s contributions are fourfold: (1) a hierarchical motion representation that separates global camera motion from object‑level dynamics, (2) an LLM‑driven semantic parsing pipeline that reliably extracts moving object names from captions, (3) dual‑condition adapters that enable a diffusion model to be steered simultaneously by pose and mask information, and (4) extensive empirical validation demonstrating semantic‑preserving extreme compression. Limitations include dependence on accurate pose estimation (which may degrade in highly dynamic or non‑rigid scenes) and the current focus on 512 × 512 resolution and relatively constrained domains (e.g., indoor/real‑estate footage). Future work could explore multi‑camera setups, real‑time encoding, and tighter integration with text‑to‑video generative models to broaden applicability. Overall, the study presents a compelling direction for next‑generation video codecs that leverage generative AI to shift the compression paradigm from pixel‑level redundancy removal to high‑level semantic synthesis.