AnchorDream: Repurposing Video Diffusion for Embodiment-Aware Robot Data Synthesis

The collection of large-scale and diverse robot demonstrations remains a major bottleneck for imitation learning, as real-world data acquisition is costly and simulators offer limited diversity and fidelity with pronounced sim-to-real gaps. While generative models present an attractive solution, existing methods often alter only visual appearances without creating new behaviors, or suffer from embodiment inconsistencies that yield implausible motions. To address these limitations, we introduce AnchorDream, an embodiment-aware world model that repurposes pretrained video diffusion models for robot data synthesis. AnchorDream conditions the diffusion process on robot motion renderings, anchoring the embodiment to prevent hallucination while synthesizing objects and environments consistent with the robot’s kinematics. Starting from only a handful of human teleoperation demonstrations, our method scales them into large, diverse, high-quality datasets without requiring explicit environment modeling. Experiments show that the generated data leads to consistent improvements in downstream policy learning, with relative gains of 36.4% in simulator benchmarks and nearly double performance in real-world studies. These results suggest that grounding generative world models in robot motion provides a practical path toward scaling imitation learning.

💡 Research Summary

The paper “AnchorDream: Repurposing Video Diffusion for Embodiment-Aware Robot Data Synthesis” presents a novel framework to address the critical bottleneck in robot imitation learning: the scarcity of large-scale, diverse, and high-quality demonstration data. Collecting real-world data is expensive, while simulators suffer from limited diversity and a significant sim-to-real gap. Existing data augmentation methods often only alter visual appearances without creating new behaviors, or they generate kinematically implausible motions due to a lack of embodiment grounding.

AnchorDream introduces a paradigm shift by repurposing large-scale, pre-trained video diffusion models as an “embodiment-aware world model.” The core innovation is decoupling trajectory generation from environment synthesis and anchoring the generative process to actual robot motion. The method starts with a small seed set of human-teleoperated demonstrations. First, it heuristically expands this set by perturbing key states (like contact points) and recombining motion segments to produce a large number of new, kinematically feasible robot trajectories.

Instead of reconstructing full environments in a simulator, AnchorDream then renders these new trajectories as clean “robot-only” motion videos, containing only the robot arm moving against a blank background. These videos serve as the primary conditioning signal. A pre-trained video diffusion model takes this motion trace, along with a language description of the task, and synthesizes photorealistic demonstration videos. The model is constrained to generate only the surrounding objects and environments in a manner consistent with the anchored robot motion, thereby preventing hallucinations of the robot’s body and ensuring physical plausibility.

To enable the generation of long-horizon sequences, the authors introduce “global trajectory conditioning.” This technique provides the diffusion model with information about the entire planned trajectory during the generation of each segment, ensuring that synthesized object placements remain compatible with the robot’s future actions, preventing scene-object mismatches over time.

Extensive experiments in simulation and on a real robot platform validate the approach. Policies trained on data synthesized by AnchorDream consistently outperform those trained on baseline datasets. The paper reports relative performance gains of 36.4% in simulation benchmarks and nearly double the success rate in real-world studies, compared to training on the original seed demonstrations or data from prior augmentation methods.

In summary, AnchorDream offers a practical and scalable path for amplifying limited robot demonstration data. It leverages the powerful visual and physical priors of internet-scale video models while firmly grounding them in real robot kinematics, bypassing the need for expensive simulation assets or explicit 3D environment reconstruction, and significantly advancing the frontier of data-driven robot learning.