Sim-and-Human Co-training for Data-Efficient and Generalizable Robotic Manipulation

Synthetic simulation data and real-world human data provide scalable alternatives to circumvent the prohibitive costs of robot data collection. However, these sources suffer from the sim-to-real visual gap and the human-to-robot embodiment gap, respectively, which limits the policy’s generalization to real-world scenarios. In this work, we identify a natural yet underexplored complementarity between these sources: simulation offers the robot action that human data lacks, while human data provides the real-world observation that simulation struggles to render. Motivated by this insight, we present SimHum, a co-training framework to simultaneously extract kinematic prior from simulated robot actions and visual prior from real-world human observations. Based on the two complementary priors, we achieve data-efficient and generalizable robotic manipulation in real-world tasks. Empirically, SimHum outperforms the baseline by up to $\mathbf{40%}$ under the same data collection budget, and achieves a $\mathbf{62.5%}$ OOD success with only 80 real data, outperforming the real only baseline by $7.1\times$. Videos and additional information can be found at \href{https://kaipengfang.github.io/sim-and-human}{project website}.

💡 Research Summary

SimHum introduces a data‑efficient co‑training framework that simultaneously leverages simulated robot actions and real‑world human observations to learn robust manipulation policies. The authors begin by highlighting two pervasive gaps in robot learning: the Sim2Real visual discrepancy (synthetic images differ from real scenes) and the Human2Robot embodiment discrepancy (human hand poses do not map directly to robot grippers). Rather than treating these gaps as problems to be explicitly bridged, SimHum treats the two data sources as complementary: simulation supplies accurate, robot‑compatible kinematic trajectories, while human demonstrations provide photorealistic visual contexts that are hard to synthesize in simulation.

Data Collection Pipeline

• Simulation data (D_sim): Generated with the exact URDF of the target robot, ensuring that the action space matches the real robot. Domain randomization (lighting, background, distractors, object poses) is applied to increase visual diversity, but the visual domain remains synthetic.
• Human data (D_hum): Collected using the same camera model and viewpoint as the robot, guaranteeing that the visual distribution aligns with the real world. Human hand poses are recorded, but the actions are not directly executable by the robot.

Both datasets cover the same set of manipulation tasks, enabling a unified pre‑training stage.

Modular Diffusion Policy Architecture
Built on the DiT diffusion policy, the network consists of:

1. A shared vision encoder that extracts raw visual tokens.
2. Domain‑specific visual adaptors (two‑layer MLPs) that transform simulation tokens and human tokens separately, isolating domain‑specific bias while feeding a consistent representation to the backbone.
3. Action encoders/decoders for each embodiment: a human encoder/decoder maps hand poses to latent tokens and back; a robot encoder/decoder does the same for robot proprioceptive states and control commands.
4. A transformer backbone that operates on the combined visual and action tokens and predicts denoising noise, following standard diffusion training.

This modular design allows the backbone to learn embodiment‑agnostic manipulation semantics, while the encoders/decoders and adaptors retain source‑specific priors.

Two‑Stage Training Paradigm

• Sim‑Human Pre‑training: Jointly train on D_sim and D_hum using a weighted loss L_total = (1‑α)L(D_sim) + αL(D_hum). The authors empirically find α = 0.5 (equal weighting) yields the best balance. This stage embeds a kinematic prior from simulation and a visual prior from human data into the respective modules.
• Real‑Robot Fine‑Tuning: Re‑assemble the policy by keeping the human visual adaptor (to preserve realistic visual priors) and the robot action encoder/decoder (to preserve transferable kinematic priors). The assembled model is then fine‑tuned on a small real‑robot dataset (as few as 80 trajectories).

Experiments
Four diverse bimanual tasks are used: Stack Bowls, Click Bell, Grab Roller, and Put Bread in Cabinet. Each task stresses different aspects such as sequential object interaction, fine‑grained precision, synchronized bimanual coordination, and long‑horizon planning.

Key results:

• With the same total data‑collection budget, reallocating half of the time to simulation and human data yields up to 40 % higher success rates compared to a baseline that only collects real robot data.
• Using only 80 real trajectories, SimHum achieves 62.5 % success on out‑of‑distribution (OOD) test scenarios, outperforming a real‑only baseline by 7.1×.
• Ablation studies confirm the importance of the co‑training ratio, the presence of domain‑specific adaptors, and the selective recombination during fine‑tuning. Removing the human visual adaptor dramatically reduces performance, underscoring the necessity of realistic visual priors.

Insights and Contributions

1. Complementarity‑Driven Co‑training: By explicitly separating and exploiting the strengths of each data source, SimHum avoids the pitfalls of naïve data pooling or heavyweight domain alignment.
2. Modular Design for Scalability: The architecture permits easy swapping of adaptors or encoders, making it adaptable to new robots, cameras, or additional data modalities.
3. Data‑Efficiency: The framework dramatically reduces the amount of costly real‑robot data needed to achieve strong generalization, a crucial step toward scalable robot learning.

Limitations and Future Work

• The current pipeline assumes identical robot kinematics and camera viewpoints across all sources; extending to heterogeneous robots or multi‑camera setups will require additional alignment mechanisms.
• Human demonstrations that involve motions impossible for the robot (e.g., finger‑level dexterity) may challenge the action encoder/decoder’s ability to translate effectively.
• Future directions include integrating automated visual domain translation (e.g., GAN‑based Sim2Real rendering) and meta‑learning approaches for cross‑embodiment action mapping, further reducing reliance on manual data alignment.

In summary, SimHum presents a principled, modular co‑training strategy that leverages the natural complementarity of simulation and human data, achieving high‑performance, data‑efficient, and OOD‑robust robotic manipulation with only a modest amount of real‑world robot experience.