A Taxonomy for Evaluating Generalist Robot Manipulation Policies

Machine learning for robot manipulation promises to unlock generalization to novel tasks and environments. But how should we measure the progress of these policies towards generalization? Evaluating and quantifying generalization is the Wild West of modern robotics, with each work proposing and measuring different types of generalization in their own, often difficult to reproduce settings. In this work, our goal is (1) to outline the forms of generalization we believe are important for robot manipulation in a comprehensive and fine-grained manner, and (2) to provide reproducible guidelines for measuring these notions of generalization. We first propose STAR-Gen, a taxonomy of generalization for robot manipulation structured around visual, semantic, and behavioral generalization. Next, we instantiate STAR-Gen with two case studies on real-world benchmarking: one based on open-source models and the Bridge V2 dataset, and another based on the bimanual ALOHA 2 platform that covers more dexterous and longer horizon tasks. Our case studies reveal many interesting insights: for example, we observe that open-source vision-language-action models often struggle with semantic generalization, despite pre-training on internet-scale language datasets. We provide videos and other supplementary material at stargen-taxonomy.github.io.

💡 Research Summary

The paper tackles the long‑standing problem of how to measure and compare the generalization abilities of robot manipulation policies. The authors argue that current literature evaluates only narrow, often incomparable aspects of generalization, making it difficult to track real‑world progress. To address this, they introduce STAR‑Gen (stylized as ‹‑Gen), a systematic taxonomy that categorizes perturbations to a base manipulation task along three policy modalities: vision, language, and actions. Perturbations are labeled as visual, semantic, or behavioral depending on whether they affect the initial image observations, the natural‑language instruction, or the optimal action distribution, respectively. By considering all possible combinations of these modalities, STAR‑Gen defines seven categories (visual‑only, semantic‑only, behavioral‑only, visual+semantic, visual+behavioral, semantic+behavioral, visual+semantic+behavioral) and a total of 22 fine‑grained axes. Visual axes include image augmentations, scene texture changes, object color changes, and viewpoint shifts. Semantic axes cover object properties (color, mass, size), language re‑phrasing, multi‑object spatial relations, human affordances, and external internet knowledge. Behavioral axes capture hidden physical parameters (mass, friction, fragility), object and scene pose changes, object morphing, robot embodiment changes, bimanual symmetry, and modifications to motion descriptors such as speed, adverbs, or action verbs.

To demonstrate the utility of the taxonomy, the authors instantiate it in two real‑world case studies. The first, BridgeV2‑‹, builds on the publicly available Bridge V2 dataset and evaluates several open‑source generalist policies (BC‑Z, RT‑Series, OpenVLA, among others) across 1,600+ trials covering all 14 evaluated axes. Results show that while policies are relatively robust to pure visual perturbations (lighting, blur, viewpoint), they struggle markedly with semantic changes that require understanding of object physical attributes or nuanced language re‑phrasing. Even large vision‑language‑action models pretrained on internet‑scale data fail to generalize when the instruction references object mass or size.

The second case study uses the bimanual ALOHA 2 platform, which supports more dexterous, longer‑horizon tasks such as two‑handed assembly and complex object manipulation. Here the authors evaluate the same taxonomy on tasks involving simultaneous visual, semantic, and behavioral shifts (e.g., changing object mass, surface friction, and instruction wording together). The findings reveal a compounded degradation: when hidden physical variables (mass, friction) are altered together with semantic changes, success rates drop dramatically, highlighting a gap in current policy training pipelines that often ignore such multi‑modal distribution shifts.

A comparative table (Table II) re‑examines existing generalization benchmarks and datasets through the lens of STAR‑Gen, showing that most prior works cover only a subset of the visual axes and rarely address semantic or behavioral dimensions. This underscores the need for a unified framework.

In conclusion, STAR‑Gen provides a comprehensive, reproducible, and extensible benchmark structure that can diagnose precisely where a manipulation policy fails to generalize. By mapping performance across well‑defined axes, researchers can target data collection, model architecture, or training objectives to the specific weaknesses identified. The taxonomy is applicable both to simulation‑based and real‑world evaluations and promises to become a standard for measuring progress toward truly generalist robot manipulators.