Learning Physics-Grounded 4D Dynamics with Neural Gaussian Force Fields

Predicting physical dynamics from raw visual data remains a major challenge in AI. While recent video generation models have achieved impressive visual quality, they still cannot consistently generate physically plausible videos due to a lack of modeling of physical laws. Recent approaches combining 3D Gaussian splatting and physics engines can produce physically plausible videos, but are hindered by high computational costs in both reconstruction and simulation, and often lack robustness in complex real-world scenarios. To address these issues, we introduce Neural Gaussian Force Field (NGFF), an end-to-end neural framework that integrates 3D Gaussian perception with physics-based dynamic modeling to generate interactive, physically realistic 4D videos from multi-view RGB inputs, achieving two orders of magnitude faster than prior Gaussian simulators. To support training, we also present GSCollision, a 4D Gaussian dataset featuring diverse materials, multi-object interactions, and complex scenes, totaling over 640k rendered physical videos (~4 TB). Evaluations on synthetic and real 3D scenarios show NGFF’s strong generalization and robustness in physical reasoning, advancing video prediction towards physics-grounded world models.

💡 Research Summary

The paper “Learning Physics‑Grounded 4D Dynamics with Neural Gaussian Force Fields” introduces Neural Gaussian Force Field (NGFF), an end‑to‑end framework that bridges visual perception and physics simulation to produce physically consistent 4‑dimensional videos from multi‑view RGB inputs. The authors identify two major shortcomings in existing approaches: (1) modern video generation models achieve impressive visual fidelity but lack explicit physics, often violating basic laws such as gravity, object permanence, and solidity; (2) recent methods that combine 3‑D Gaussian splatting with traditional physics engines enforce physical consistency but are computationally prohibitive and struggle with complex, real‑world multi‑object interactions.

NGFF addresses these gaps by (i) reconstructing a scene into object‑aware 3‑D Gaussians using a feed‑forward transformer pipeline, (ii) segmenting the Gaussians into distinct objects with SAM2 and refining them via DiffSplat to handle occlusions, (iii) learning object‑centric force fields through a neural operator built on a relational graph, and (iv) integrating the predicted force fields with a second‑order ODE solver to generate continuous, physically plausible trajectories. The reconstructed Gaussians are rendered by differentiable Gaussian splatting, enabling high‑quality multi‑view video synthesis while preserving physical realism.

The reconstruction module tokenizes images with DINOv2, processes them through an L‑layer Alternating‑Attention Transformer, and predicts camera poses, Gaussian centers, and per‑Gaussian attributes (color, radius, orientation). Object‑level features are extracted via PointNet, and each object’s state is represented by semantic embeddings, zero‑order quantities (point cloud, center of mass, orientation) and first‑order quantities (velocities, angular velocities). A relational graph encodes contact relationships; a DeepONet‑style neural operator computes a global force vector for each object by aggregating encoded neighbor information. For deformable bodies, a separate network Φ predicts point‑wise stress fields using a Contact Area Mask, and the final force is the sum of global and local components.

Training proceeds in two stages. First, the reconstruction network is pretrained on the WildRGBD dataset, freezing geometry‑related heads while fine‑tuning the splatter head with RGB‑depth consistency losses. Second, the dynamics network is trained on synthetic Material Point Method (MPM) simulations covering a wide range of rigid and soft materials, optimizing mean‑squared error between predicted Gaussian configurations and ground‑truth trajectories. This decoupled strategy leverages abundant visual data and accurate physics simulations without destabilizing the joint optimization.

To evaluate NGFF, the authors release GSCollision, a 4‑D Gaussian dataset comprising 640,000 rendered physical videos (~4 TB) that feature diverse materials, multi‑object collisions, sliding, containment, and real‑world backgrounds from WildRGBD. Experiments cover dynamic prediction accuracy, physical consistency (energy conservation, correct collision response), novel‑view synthesis, background‑agnostic generation, and sim‑to‑real transfer. NGFF outperforms state‑of‑the‑art baselines such as Pointformer, VEO3, NVIDIA Cosmos, and PhysGen3D, achieving more than 30 % lower position/velocity error and a two‑order‑of‑magnitude speedup (≈0.01 s per frame). Importantly, the model generalizes to out‑of‑distribution configurations—different object counts, arrangements, and material properties—while maintaining stable predictions.

The paper’s contributions can be summarized as:

1. A unified pipeline that couples fast feed‑forward Gaussian reconstruction with neural force‑field dynamics, enabling real‑time 4‑D video prediction.
2. An explicit force‑field formulation that captures both rigid‑body interactions and soft‑body deformations via global and local components, learned through a graph‑based neural operator.
3. A differentiable ODE integration that provides continuous trajectories and allows end‑to‑end training with rendering losses.
4. The GSCollision dataset, the largest collection of physics‑grounded Gaussian videos to date, facilitating future research on visual‑physics learning.
5. Demonstrated robustness in sim‑to‑real transfer, novel‑view synthesis, and interactive “what‑if” scenarios through force‑prompted control.

Overall, NGFF represents a significant step toward physics‑grounded world models that can both perceive complex scenes from raw images and predict their future evolution with high fidelity and efficiency. Its modular design, strong generalization, and real‑time performance open avenues for applications in robotics, AR/VR, interactive content creation, and scientific simulation where both visual realism and physical correctness are essential.