RDT2: Exploring the Scaling Limit of UMI Data Towards Zero-Shot Cross-Embodiment Generalization

Vision-Language-Action (VLA) models hold promise for generalist robotics but currently struggle with data scarcity, architectural inefficiencies, and the inability to generalize across different hardware platforms. We introduce RDT2, a robotic foundation model built upon a 7B parameter VLM designed to enable zero-shot deployment on novel embodiments for open-vocabulary tasks. To achieve this, we collected one of the largest open-source robotic datasets–over 10,000 hours of demonstrations in diverse families–using an enhanced, embodiment-agnostic Universal Manipulation Interface (UMI). Our approach employs a novel three-stage training recipe that aligns discrete linguistic knowledge with continuous control via Residual Vector Quantization (RVQ), flow-matching, and distillation for real-time inference. Consequently, RDT2 becomes one of the first models that simultaneously zero-shot generalizes to unseen objects, scenes, instructions, and even robotic platforms. Besides, it outperforms state-of-the-art baselines in dexterous, long-horizon, and dynamic downstream tasks like playing table tennis. See https://rdt-robotics.github.io/rdt2/ for more information.

💡 Research Summary

RDT2 presents a comprehensive solution to the long‑standing challenges of scaling Vision‑Language‑Action (VLA) models for robotics: data scarcity, multimodal action representation, and real‑time inference. The authors first redesign the Universal Manipulation Interface (UMI) hardware, replacing 3‑D‑printed components with CNC‑machined nylon‑glass‑fiber parts, adding infrared‑light SLAM tracking, and introducing a compact parallel‑jaw gripper. These upgrades enable the deployment of roughly 100 UMI devices across more than 100 households, yielding over 10 000 hours of diverse, in‑the‑wild manipulation demonstrations that include deformable objects, fluids, and high‑speed dynamics.

The model architecture builds on a 7 B pretrained vision‑language model (Qwen2.5‑VL) and adds specialized action heads. Continuous robot actions are first discretized using Residual Vector Quantization (RVQ): a 1‑D CNN encoder maps action chunks to latent vectors, which are iteratively quantized through multiple codebooks, producing token indices while preserving reconstruction fidelity via a combined quantization‑reconstruction loss. Stage 1 trains the VLM backbone with a cross‑entropy loss on these tokens, preserving the pretrained discrete probability knowledge.

Stage 2 freezes the backbone and trains a diffusion‑based action expert using a flow‑matching loss, which models the diffusion dynamics as a continuous vector field, allowing efficient sampling of multimodal action distributions without the slow convergence typical of diffusion policies.

Stage 3 addresses the real‑time requirement by distilling the multi‑step expert into a single‑step generator. The distillation loss aligns the generator’s output distribution with that of the expert, enabling inference at >30 Hz, suitable for on‑policy robotic control.

Evaluation focuses on four axes of zero‑shot generalization: unseen objects, novel scenes, new natural‑language instructions, and entirely new robot embodiments (different kinematics and degrees of freedom). RDT2 consistently outperforms state‑of‑the‑art baselines such as π‑0 and π‑0.5 across these axes, with particularly strong results on tasks involving deformable object manipulation, long‑horizon planning, and high‑dynamic activities like table tennis. Ablation studies confirm the necessity of each training stage, and scaling experiments demonstrate a predictable performance gain when both model size and dataset volume increase, establishing a clear scaling law for robotic foundation models.

In summary, RDT2 delivers (1) a massive, heterogeneous real‑world robotic dataset enabled by a robust, embodiment‑agnostic UMI; (2) a hybrid discretization‑diffusion training pipeline that leverages the strengths of both discrete token alignment and continuous action modeling; and (3) an efficient single‑step inference engine through knowledge distillation. This integrated approach sets a new benchmark for zero‑shot cross‑embodiment generalization and provides a practical roadmap for future larger‑scale robot foundation models.