Correspondence-Free, Function-Based Sim-to-Real Learning for Deformable Surface Control

This paper presents a correspondence-free, function-based sim-to-real learning method for controlling deformable freeform surfaces. Unlike traditional sim-to-real transfer methods that strongly rely on marker points with full correspondences, our approach simultaneously learns a deformation function space and a confidence map – both parameterized by a neural network – to map simulated shapes to their real-world counterparts. As a result, the sim-to-real learning can be conducted by input from either a 3D scanner as point clouds (without correspondences) or a motion capture system as marker points (tolerating missed markers). The resultant sim-to-real transfer can be seamlessly integrated into a neural network-based computational pipeline for inverse kinematics and shape control. We demonstrate the versatility and adaptability of our method on both vision devices and across four pneumatically actuated soft robots: a deformable membrane, a robotic mannequin, and two soft manipulators.

💡 Research Summary

This paper introduces a correspondence‑free, function‑based sim‑to‑real transfer framework for controlling highly deformable free‑form surfaces of soft robots. Traditional sim‑to‑real approaches rely on dense marker correspondences, which limits their applicability to 3D scanners (producing unstructured point clouds) and makes them fragile to occlusions in motion‑capture (MoCap) data. The authors address these limitations by (1) representing each simulated surface with a compact B‑spline model, extracting a low‑dimensional set of control points (S_c) as the shape descriptor, and (2) learning a continuous spatial warping function (\Phi(p)) that maps any point on the simulated surface to its corrected real‑world counterpart.

The warping function is built from an affine term and a sum of Gaussian radial basis functions (RBFs): (\Phi(p)=\alpha_0 + A p + \sum_i \beta_i \exp(-c|p-q_i|^2)). Kernel centers (q_i) are sampled uniformly over the UV domain of the B‑spline surface, ensuring coverage of the entire geometry. The coefficients (\gamma =