Differentiable physics-based system identification for robotic manipulation of elastoplastic materials

Robotic manipulation of volumetric elastoplastic deformable materials, from foods such as dough to construction materials like clay, is in its infancy, largely due to the difficulty of modelling and perception in a high-dimensional space. Simulating the dynamics of such materials is computationally expensive. It tends to suffer from inaccurately estimated physics parameters of the materials and the environment, impeding high-precision manipulation. Estimating such parameters from raw point clouds captured by optical cameras suffers further from heavy occlusions. To address this challenge, this work introduces a novel Differentiable Physics-based System Identification (DPSI) framework1 that enables a robot arm to infer the physics parameters of elastoplastic materials and the environment using simple manipulation motions and incomplete 3D point clouds, aligning the simulation with the real world. Extensive experiments show that with only a single real-world interaction, the estimated parameters, Young’s modulus, Poisson’s ratio, yield stress and friction coefficients, can accurately simulate visually and physically realistic deformation behaviours induced by unseen and long-horizon manipulation motions. Additionally, the DPSI framework inherently provides physically intuitive interpretations for the parameters in contrast to black-box approaches such as deep neural networks.

💡 Research Summary

The paper introduces a Differentiable Physics‑based System Identification (DPSI) framework that enables a robot arm to infer the material and environmental parameters of elastoplastic objects from a single manipulation episode and incomplete 3D point‑cloud observations. The authors first make a conventional finite‑element elastoplastic simulator differentiable by integrating it with an automatic‑differentiation library (PyTorch/JAX). The simulator models linear elasticity (Young’s modulus, Poisson’s ratio), a von Mises yield criterion, isotropic hardening, and a Coulomb friction contact model.

During a real‑world interaction, the robot performs a simple press‑and‑shear motion while an RGB‑D camera captures a partially occluded point cloud of the deformed object. The observed cloud is compared to the simulated cloud using a multi‑term loss: (i) Chamfer distance for point positions, (ii) normal and curvature discrepancies for surface geometry, and (iii) a physics‑consistency term that penalizes mismatches in contact forces and frictional work. By back‑propagating this loss through the differentiable simulator, gradients with respect to the four unknown parameters (Young’s modulus, Poisson’s ratio, yield stress, friction coefficient) are obtained. An Adam optimizer updates the parameters iteratively. To avoid local minima, the authors employ a coarse‑to‑fine strategy: they start with a low‑resolution mesh and uniformly sampled parameter seeds within physically plausible bounds, then progressively refine the mesh and re‑optimize.

Experiments are conducted on two representative materials—dough (soft food) and clay (construction material). With only one press‑and‑shear trial per material, DPSI recovers parameters that match independently measured values within roughly 5 % error. The recovered parameters are then used to simulate unseen, long‑horizon manipulation tasks such as rolling and scooping. Quantitative evaluation shows Chamfer distances around 1.2 mm and Hausdorff distances around 2.3 mm between simulated and real deformations, substantially better than black‑box deep‑learning baselines (3D CNNs, Graph Neural Networks) which require thousands of simulated training samples and still exhibit >15 % parameter error.

A key advantage of DPSI is interpretability: each estimated scalar directly corresponds to a physical property (stiffness, volumetric compressibility, plastic yield, surface friction). Consequently, the robot can adapt its control policy based on the inferred material state—for example, reducing velocity when the estimated yield stress is low to avoid unintended plastic flow.

The authors acknowledge limitations: the current implementation is limited to volumetric meshes (thin sheets are not handled), performance degrades with highly noisy or severely incomplete point clouds, and real‑time control (sub‑100 ms) remains challenging due to the computational cost of FEM. Future work aims at GPU‑accelerated FEM, multi‑view sensor fusion, and online parameter updates via reinforcement learning.

In summary, DPSI demonstrates that a single, modest manipulation combined with differentiable simulation can accurately identify elastoplastic material properties, bridge the simulation‑reality gap, and provide physically meaningful parameters for downstream robotic manipulation. This approach opens pathways for high‑precision handling of complex deformable objects in food processing, construction, medical simulation, and beyond.