VS3R: Robust Full-frame Video Stabilization via Deep 3D Reconstruction

Video stabilization aims to mitigate camera shake but faces a fundamental trade-off between geometric robustness and full-frame consistency. While 2D methods suffer from aggressive cropping, 3D techniques are often undermined by fragile optimization pipelines that fail under extreme motions. To bridge this gap, we propose VS3R, a framework that synergizes feed-forward 3D reconstruction with generative video diffusion. Our pipeline jointly estimates camera parameters, depth, and masks to ensure all-scenario reliability, and introduces a Hybrid Stabilized Rendering module that fuses semantic and geometric cues for dynamic consistency. Finally, a Dual-Stream Video Diffusion Model restores disoccluded regions and rectifies artifacts by synergizing structural guidance with semantic anchors. Collectively, VS3R achieves high-fidelity, full-frame stabilization across diverse camera models and significantly outperforms state-of-the-art methods in robustness and visual quality.

💡 Research Summary

**
VS3R tackles the long‑standing dilemma in video stabilization between geometric robustness and full‑frame consistency. Traditional 2‑D approaches rely on planar transformations (affine, homography, mesh warping) and inevitably resort to aggressive cropping to hide dis‑occluded regions, which sacrifices field‑of‑view (FoV) and introduces temporal flicker. Recent learning‑based 2‑D methods improve warping but still lack explicit 3‑D scene constraints, leading to structural distortions under parallax. Conversely, emerging 3‑D pipelines (NeRF, 3‑D Gaussian Splatting) adopt a reconstruct‑and‑render paradigm but depend heavily on Structure‑from‑Motion (SfM) for camera pose estimation. SfM collapses in degenerate cases such as pure rotation, severe motion blur, or rapid motion, causing tracking failures and scale drift. Moreover, these methods struggle with dynamic objects and often produce incomplete boundaries, forcing additional cropping or post‑processing.

VS3R proposes a unified “reconstruct‑smooth‑refine” framework that eliminates the reliance on SfM and integrates generative diffusion for full‑frame restoration. The pipeline consists of three core stages:

1. Deep 3‑D Reconstruction – Using the feed‑forward VGGT4D model, VS3R jointly predicts per‑frame camera intrinsics/extrinsics, dense depth maps, and semantic dynamic masks from an uncalibrated video. A sliding‑window scheme processes long sequences while limiting memory growth, and a temporal Gaussian filter smooths the raw pose trajectory to suppress global drift.

2. Hybrid Stabilized Rendering (HSR) – VS3R refines the raw semantic masks by fusing them with a geometric mask derived from the discrepancy between observed optical flow (computed by a pretrained RAFT model) and rigid‑flow induced by the estimated ego‑motion. The union of these masks yields a robust dynamic‑region indicator. Static regions are aggregated across the temporal window to build a multi‑view point cloud, while dynamic regions are kept frame‑wise to preserve non‑rigid motion. The smoothed camera pose is then used to re‑project the composite point cloud, producing an initially stabilized frame that is geometrically coherent but may contain cropped borders, dis‑occlusion holes, and sampling noise.

3. Dual‑Stream Video Diffusion Model (DVDM) – To achieve full‑frame, temporally consistent output, VS3R introduces a diffusion model with two conditioning streams. The first stream feeds the rendered frames as spatial‑temporal priors; the second stream injects a fixed textual embedding that serves as a semantic anchor, guiding the model to maintain consistent visual style and content across frames. By cross‑attending between the streams, the diffusion process fills dis‑occluded regions, removes rendering artifacts, and restores the original FoV, yielding high‑fidelity stabilized video.

Extensive experiments on public benchmarks (e.g., NUS, DeepStab) and a custom dataset featuring extreme rotations, blur, and dynamic objects demonstrate that VS3R outperforms state‑of‑the‑art 2‑D methods (RobustL1, DIFRINT) and 3‑D methods (RStab, GaVS). Quantitatively, VS3R achieves higher PSNR/SSIM scores and a substantially larger FoV retention ratio (≈15% improvement). Subjective user studies confirm that viewers perceive VS3R videos as smoother and more natural, with fewer visual artifacts.

Key contributions include: (1) a fully feed‑forward 3‑D reconstruction pipeline that is robust to degenerate motion; (2) a hybrid mask strategy that accurately separates static and dynamic scene components; (3) a dual‑stream diffusion refinement that restores full‑frame content without aggressive cropping. Limitations are noted: the current system assumes a single camera with fixed intrinsics, and real‑time performance on high‑resolution (4K) video remains an open challenge due to the computational cost of point‑cloud rendering and diffusion inference.

Future work may explore multi‑camera synchronization, lightweight diffusion architectures for real‑time deployment, and richer semantic conditioning (e.g., style transfer or user‑specified prompts) to enable customizable stabilization. Overall, VS3R represents a significant step forward by marrying deep 3‑D reconstruction with generative video diffusion, delivering robust, full‑frame video stabilization even under the most demanding motion conditions.