SoMA: A Real-to-Sim Neural Simulator for Robotic Soft-body Manipulation

Simulating deformable objects under rich interactions remains a fundamental challenge for real-to-sim robot manipulation, with dynamics jointly driven by environmental effects and robot actions. Existing simulators rely on predefined physics or data-driven dynamics without robot-conditioned control, limiting accuracy, stability, and generalization. This paper presents SoMA, a 3D Gaussian Splat simulator for soft-body manipulation. SoMA couples deformable dynamics, environmental forces, and robot joint actions in a unified latent neural space for end-to-end real-to-sim simulation. Modeling interactions over learned Gaussian splats enables controllable, stable long-horizon manipulation and generalization beyond observed trajectories without predefined physical models. SoMA improves resimulation accuracy and generalization on real-world robot manipulation by 20%, enabling stable simulation of complex tasks such as long-horizon cloth folding.

💡 Research Summary

The paper introduces SoMA, a Real‑to‑Sim neural simulator designed for robotic manipulation of soft, deformable objects such as cloth or flexible polymers. The core idea is to reconstruct the object from multi‑view RGB videos as a set of 3‑D Gaussian splats (GS) and to evolve these splats over time using a force‑driven neural dynamics model that is directly conditioned on the robot’s joint‑space actions.

Data acquisition and preprocessing
Three calibrated cameras capture synchronized RGB streams while the robot records its joint angles at each timestep. Standard multi‑view geometry (e.g., COLMAP) provides camera poses, and a recent Gaussian‑splatting pipeline (VGGT, AnySplat, etc.) reconstructs the object geometry as a cloud of splats in the camera coordinate frame.

Unified simulation space
To bridge the gap between visual reconstruction, robot kinematics, and physical reference frames, the authors estimate a global scale factor by aligning known dimensions (e.g., robot end‑effector size) with the reconstructed point cloud. A similarity transform (scale s, rotation R, translation t) maps the camera‑based reconstruction into a simulation space that shares the robot base frame. The gravity direction is resolved by fitting the supporting table plane and comparing its normal with the camera viewing direction.

Hierarchical Gaussian‑splat representation
Each splat gᵢ = {xᵢ, Σᵢ, mᵢ, aᵢ} encodes position, anisotropic covariance, mass, and auxiliary physical attributes. Splats are recursively clustered to form a multi‑level graph: leaf nodes capture fine‑grained geometry, while higher‑level clusters aggregate mass, centroid, and attribute averages. This hierarchy enables the model to propagate global motions efficiently while preserving local deformations.

Force‑driven neural dynamics
The dynamics are not a pure state‑to‑state regression; instead, explicit forces act on each node. Environmental forces consist of gravity and a support force applied when a splat is close to the supporting plane (distance dᵢ < τ). Robot‑induced forces are computed via an interaction graph that connects robot control points (derived from forward kinematics of the joint vector qₜ) and nearby splats. A graph neural network Φθ predicts the robot force fᵣₒbᵢ for each splat based on its past states, neighboring robot points, and the gripper opening cₜ. The total force fᵢ = f_envᵢ + f_robᵢ is fed into a hierarchical dynamics module ψθ, which predicts linear and angular velocities (vᵢ, ωᵢ). These velocities update positions and covariances through simple integration, and the updates are propagated down the hierarchy.

Multi‑resolution training for long horizons
Training proceeds in two stages. Stage 1 uses a large temporal stride (k frames) to capture coarse, long‑range motion patterns, encouraging the network to learn stable global dynamics. Stage 2 reduces the stride to 1 frame, focusing on fine‑grained contact, occlusion handling, and high‑frequency deformation. The loss combines an image‑space reconstruction term (L2 between rendered splat images and ground‑truth RGB) with physics‑inspired consistency terms (e.g., momentum conservation, force regularization). This blended supervision mitigates drift and collapse that often plague long‑horizon neural simulators.

Experimental evaluation
The authors evaluate SoMA on public deformable‑object benchmarks and a newly collected real‑world dataset featuring cloth folding, pulling, and twisting tasks. Quantitatively, SoMA achieves a PSNR of 27.57 dB, surpassing prior Gaussian‑splat dynamics baselines (≈ 26.5 dB). More importantly, when the simulated trajectories are replayed on the real robot, the positional error drops by roughly 20 % compared to existing methods. The system also generalizes to unseen manipulation sequences (e.g., a new folding pattern not present in training) and maintains stable simulation for over 30 seconds of continuous contact, a regime where many learned simulators diverge.

Ablation studies confirm the necessity of each component: removing the robot‑conditioned mapping degrades accuracy dramatically, omitting the force‑driven formulation leads to unrealistic deformations, and training without the multi‑resolution schedule results in rapid drift.

Limitations and future work
SoMA currently relies solely on RGB and joint data; integrating tactile or force‑sensor feedback could improve contact fidelity. Handling heterogeneous material properties (e.g., multi‑layered fabrics) would require additional property supervision. Real‑time performance is acceptable on high‑end GPUs but would benefit from model compression for deployment on embedded platforms.

Conclusion
SoMA bridges the gap between physics‑based simulators (which need hand‑tuned parameters) and pure data‑driven models (which lack control conditioning). By representing deformable objects as hierarchical Gaussian splats and driving their evolution with robot‑conditioned forces, SoMA delivers accurate, stable, and generalizable simulations of soft‑body manipulation, enabling better data synthesis, policy learning, and transfer from simulation to real robots.