SkeletonGaussian: Editable 4D Generation through Gaussian Skeletonization

4D generation has made remarkable progress in synthesizing dynamic 3D objects from input text, images, or videos. However, existing methods often represent motion as an implicit deformation field, which limits direct control and editability. To address this issue, we propose SkeletonGaussian, a novel framework for generating editable dynamic 3D Gaussians from monocular video input. Our approach introduces a hierarchical articulated representation that decomposes motion into sparse rigid motion explicitly driven by a skeleton and fine-grained non-rigid motion. Concretely, we extract a robust skeleton and drive rigid motion via linear blend skinning, followed by a hexplane-based refinement for non-rigid deformations, enhancing interpretability and editability. Experimental results demonstrate that SkeletonGaussian surpasses existing methods in generation quality while enabling intuitive motion editing, establishing a new paradigm for editable 4D generation. Project page: https://wusar.github.io/projects/skeletongaussian/

💡 Research Summary

SkeletonGaussian introduces a novel pipeline for generating editable dynamic 3D Gaussian models (i.e., 4D objects) from a single monocular video. The authors identify two major shortcomings of existing 4D generation approaches that rely on implicit deformation fields: (1) the deformation parameters grow quadratically or even cubically with sequence length, making long‑duration generation memory‑intensive, and (2) editing such fields requires costly re‑optimization, offering little real‑time control. To overcome these issues, the paper proposes a hierarchical, skeleton‑driven representation that separates motion into a sparse rigid component and a fine‑grained non‑rigid component.

The pipeline consists of three stages. First, a static 3D Gaussian model is built from the video’s middle frame using state‑of‑the‑art Gaussian splatting methods (e.g., DreamGaussian). The point cloud is converted to a mesh via occupancy fields and marching cubes, after which a generic rigging system (UniRig by default, with Coverage Axis++ as an ablation) extracts a kinematic tree of joints. This skeleton is category‑agnostic and provides a compact control scaffold.

Second, the rigid motion of the object is modeled with Linear Blend Skinning (LBS). For each Gaussian point, the K‑nearest joints are identified and inverse‑distance weights are assigned. Joint transformations are computed by forward kinematics from per‑frame quaternion poses θ_t and a root translation t_o. The weighted sum of joint transforms yields a per‑point matrix T_i, which updates both the position and orientation of the Gaussian. The pose tensor θ ∈ ℝ^{T×B×4} is optimized directly against a multi‑view Score Distillation Sampling (MV‑SDS) loss and a photometric consistency loss, ensuring that the rendered views match the input video. Because the number of pose parameters scales linearly with the number of joints B and frames T, the representation is far more memory‑efficient than dense deformation fields.

Third, to capture subtle deformations such as cloth wrinkles or skin elasticity, a non‑rigid refinement stage is added. The authors employ a hexplane structure—a set of six planar deformation fields covering 3D space—combined with a lightweight MLP. This hybrid field, denoted F_nr, refines the LBS‑deformed Gaussian without altering the skeleton parameters. Training in this stage focuses solely on the Gaussian attributes and the hexplane‑MLP, again using MV‑SDS and photometric losses. The hexplane design reduces the parameter count compared to pure MLP grids while preserving high‑frequency detail.

All three stages share the same differentiable Gaussian rasterizer and loss functions, enabling end‑to‑end optimization. Experiments on the Consistent4D benchmark demonstrate that SkeletonGaussian achieves higher PSNR (≈0.8–1.2 dB gain) and better perceptual metrics (LPIPS, FID) than prior dynamic Gaussian methods such as Dynamic‑GS and SC‑GS. Qualitative results show superior handling of large rotations, complex clothing dynamics, and intricate non‑rigid motions.

A key contribution is real‑time editability: users can manipulate joint quaternions or root translation directly, instantly observing the effect on the rendered 4D sequence. The resulting motion can be exported in standard skeletal formats (BVH, FBX), facilitating seamless integration with existing animation pipelines like Blender.

The paper also discusses limitations. Skeleton extraction may be less reliable for highly irregular or mechanical objects, and the fixed hexplane may struggle with extreme deformations (e.g., tearing). Future work includes learning adaptive skinning weights, dynamic hexplane hierarchies, and extending the framework to conditional generation scenarios such as text‑to‑4D or image‑to‑4D.

In summary, SkeletonGaussian provides a compelling solution to the editability problem in 4D generation by marrying a skeleton‑driven LBS backbone with a hexplane‑based non‑rigid refinement, delivering high‑quality, parameter‑efficient, and interactively editable dynamic 3D Gaussian models. This approach opens new avenues for content creation in games, film, and interactive AR/VR applications.