LIBERO-X: Robustness Litmus for Vision-Language-Action Models

Reliable benchmarking is critical for advancing Vision-Language-Action (VLA) models, as it reveals their generalization, robustness, and alignment of perception with language-driven manipulation tasks. However, existing benchmarks often provide limited or misleading assessments due to insufficient evaluation protocols that inadequately capture real-world distribution shifts. This work systematically rethinks VLA benchmarking from both evaluation and data perspectives, introducing LIBERO-X, a benchmark featuring: 1) A hierarchical evaluation protocol with progressive difficulty levels targeting three core capabilities: spatial generalization, object recognition, and task instruction understanding. This design enables fine-grained analysis of performance degradation under increasing environmental and task complexity; 2) A high-diversity training dataset collected via human teleoperation, where each scene supports multiple fine-grained manipulation objectives to bridge the train-evaluation distribution gap. Experiments with representative VLA models reveal significant performance drops under cumulative perturbations, exposing persistent limitations in scene comprehension and instruction grounding. By integrating hierarchical evaluation with diverse training data, LIBERO-X offers a more reliable foundation for assessing and advancing VLA development.

💡 Research Summary

The paper introduces LIBERO‑X, a next‑generation benchmark designed to rigorously evaluate Vision‑Language‑Action (VLA) models for robotic manipulation. The authors argue that existing benchmarks such as LIBERO, SimpleEnv, and their extensions suffer from two fundamental shortcomings: (1) evaluation protocols typically manipulate a single factor (e.g., spatial jitter) in isolation, failing to capture the coupled, multi‑dimensional distribution shifts that occur in real‑world deployments; and (2) training datasets are narrowly scoped, pairing each scene with a single task and a handful of nearly identical demonstrations, which encourages memorization rather than genuine skill acquisition.

To address these issues, LIBERO‑X combines a hierarchical, multi‑level evaluation framework with a high‑diversity, human‑teleoperated training set. The evaluation hierarchy consists of five difficulty levels, each adding new perturbation dimensions on top of the previous ones:

• Level 1‑2 – Incremental spatial perturbations, ranging from minor positional jitter to broader randomization, probing basic spatial generalization.
• Level 3 – Scene topology reconstruction (e.g., swapping target and placement locations) that breaks fixed spatial associations learned during training.
• Level 4 – Visual attribute variation, including changes in color, texture, size, and the introduction of unseen, confounding objects, testing object‑recognition robustness.
• Level 5 – Semantic‑equivalent instruction reformulation, where natural‑language commands are paraphrased while preserving intent, assessing language‑vision‑action alignment.

Each level is annotated with a set of fine‑grained multi‑label metrics (interaction type, sub‑task count, spatial relation, object attributes), enabling precise diagnostics of failure modes across manipulation dimensions.

On the data side, the authors collected 2,520 demonstrations across 600 tasks and 100 scenes via human teleoperation in MuJoCo. Crucially, a single scene now supports multiple distinct tasks, and each task is demonstrated with varied object attributes (color, texture, size) and spatial relations (left/right, near/far, front/behind). This design dramatically expands the scene‑task‑trajectory distribution compared with the original LIBERO, mitigating over‑fitting and providing a richer learning substrate.

The benchmark is evaluated on several representative VLA models, including discrete‑token approaches (OpenVLA, RT‑2) and continuous‑regression methods (RoboFlamingo, CogACT). All models are fine‑tuned on the new training set and then tested across the five levels. Results reveal a systematic performance drop as difficulty increases: while models achieve high success rates on Levels 1‑2, their performance deteriorates sharply at Level 3 (topology changes) and even more so at Levels 4‑5, where visual attribute shifts and instruction paraphrases cause substantial failures. Error analyses, enabled by the multi‑label annotations, pinpoint that current architectures rely heavily on low‑level visual cues (e.g., absolute object positions and colors) and lack robust high‑level reasoning about spatial relationships and semantic equivalence.

The contributions of the work are threefold:

1. LIBERO‑X benchmark – a comprehensive, progressive evaluation suite with multi‑label diagnostics that jointly perturbs spatial layouts, object properties, and language semantics.
2. High‑diversity training dataset – collected via human teleoperation, providing unprecedented scene and task variety for VLA model training.
3. Empirical insights – extensive experiments exposing critical weaknesses in state‑of‑the‑art VLA models, especially in scene comprehension and instruction grounding under compounded distribution shifts.

By integrating richer training data with a systematic, multi‑dimensional test hierarchy, LIBERO‑X offers a more faithful measure of VLA model robustness and generalization, guiding future research toward architectures that can truly operate reliably in complex, dynamic robotic environments.