Benchmarking Affordance Generalization with BusyBox

Vision-Language-Action (VLA) models have been attracting the attention of researchers and practitioners thanks to their promise of generalization. Although single-task policies still offer competitive performance, VLAs are increasingly able to handle commands and environments unseen in their training set. While generalization in vision and language space is undoubtedly important for robust versatile behaviors, a key meta-skill VLAs need to possess is affordance generalization – the ability to manipulate new objects with familiar physical features. In this work, we present BusyBox, a physical benchmark for systematic semi-automatic evaluation of VLAs’ affordance generalization. BusyBox consists of 6 modules with switches, sliders, wires, buttons, a display, and a dial. The modules can be swapped and rotated to create a multitude of BusyBox variations with different visual appearances but the same set of affordances. We empirically demonstrate that generalization across BusyBox variants is highly challenging even for strong open-weights VLAs such as $π_{0.5}$ and GR00T-N1.6. To encourage the research community to evaluate their own VLAs on BusyBox and to propose new affordance generalization experiments, we have designed BusyBox to be easy to build in most robotics labs. We release the full set of CAD files for 3D-printing its parts as well as a bill of materials for (optionally) assembling its electronics. We also publish a dataset of language-annotated demonstrations that we collected using the common bimanual Mobile Aloha robot on the canonical BusyBox configuration. All of the released materials are available at https://microsoft.github.io/BusyBox.

💡 Research Summary

The paper introduces BusyBox, a novel physical benchmark designed to evaluate the affordance‑generalization capability of Vision‑Language‑Action (VLA) models. BusyBox consists of six interchangeable modules—buttons, sliders, switches, wires, a display, and a dial—each representing a basic affordance commonly found in everyday devices. By allowing the modules to be swapped and rotated, the benchmark can generate a large family of visually distinct configurations while preserving the same set of manipulable functions. This property enables a direct test of whether a robot can transfer the abstract instruction “press the button” or “rotate the knob to position 4” to a novel visual layout it has never encountered.

The authors provide a complete open‑source package: 3‑D printable CAD files, a bill of materials, and optional electronics that continuously broadcast the state of every control element at 10 Hz via a Raspberry Pi 0. This instrumentation facilitates automatic labeling of demonstration trajectories and real‑time monitoring without requiring manual annotation.

To create a dataset for training and evaluation, the authors tele‑operated a dual‑arm Mobile Aloha robot to collect 1,993 manipulation demonstrations on a canonical BusyBox configuration. Each episode begins from a randomly sampled initial state (varying slider positions, knob value, switch states, wire insertion, and the pose of the box and robot) that deliberately does not satisfy the goal instruction, ensuring that the robot must actively manipulate the device. Demonstrations are constrained to be efficient (≤20 s for most tasks, ≤45 s for wire insertion) and include both unilateral and bimanual actions.

Two state‑of‑the‑art open‑weight VLA models—π₀.₅ and GR00T‑N1.6—are fine‑tuned on the collected dataset. The authors then evaluate the fine‑tuned models on three BusyBox configurations: (a) the canonical layout used for data collection, (b) a semi‑shuffled layout where three modules are repositioned and one is rotated, and (c) a fully‑shuffled layout where all five manipulable modules are moved or rotated. Success is defined by the electronic instrumentation confirming that the target state has been reached.

Results show that both models achieve high success rates (>80 %) on the canonical configuration, indicating that the dataset is sufficient for learning in‑distribution tasks. However, performance drops dramatically on the semi‑shuffled layout (≈45 % success) and further declines on the fully‑shuffled layout (≈30 % success). The degradation is especially pronounced for tasks involving modules whose spatial positions have changed (buttons, sliders, switches). These findings demonstrate that current VLA models rely heavily on visual layout cues and struggle to abstract the underlying affordances independent of appearance.

Beyond the presented experiments, BusyBox is positioned as a versatile platform for a wide range of research questions. Its modularity supports tests of spatial reasoning (“pull the second wire from the left”), language‑based corrective feedback (“press the blue button instead of the red one”), and human‑robot interaction scenarios inspired by cooperative games such as “Keep Talking and Nobody Explodes.” The benchmark thus fills a gap in the robotics community: a systematic, reproducible, and physically grounded test of affordance generalization, a meta‑skill essential for robots operating in human‑centric environments.

In summary, the paper contributes (1) the BusyBox hardware benchmark with full open‑source designs, (2) a curated language‑annotated demonstration dataset covering eight affordance families, and (3) an evaluation protocol with baseline results showing that even the strongest open‑weight VLA models are far from achieving robust affordance generalization. BusyBox is released publicly, inviting the community to benchmark, extend, and improve VLA models toward more human‑like, function‑centric manipulation capabilities.