Benchmarking the Generality of Vision-Language-Action Models

Generalist multimodal agents are expected to unify perception, language, and control - operating robustly across diverse real world domains. However, current evaluation practices remain fragmented across isolated benchmarks, making it difficult to assess whether today’s foundation models truly generalize beyond their training distributions. We introduce MultiNet v1.0, a unified benchmark for measuring the cross domain generality of vision language models (VLMs) and vision language action models (VLAs) across six foundational capability regimes. Visual grounding, spatial reasoning, tool use, physical commonsense, multi agent coordination, and continuous robot control. Evaluating GPT 5, Pi0, and Magma, we find that no model demonstrates consistent generality. All exhibit substantial degradation on unseen domains, unfamiliar modalities, or cross domain task shifts despite strong performance within their training distributions.These failures manifest as modality misalignment, output format instability, and catastrophic knowledge degradation under domain transfer.Our findings reveal a persistent gap between the aspiration of generalist intelligence and the actual capabilities of current foundation models.MultiNet v1.0 provides a standardized evaluation substrate for diagnosing these gaps and guiding the development of future generalist agents.Code, data, and leaderboards are publicly available.

💡 Research Summary

The paper addresses a critical gap in the evaluation of vision‑language‑action (VLA) systems: existing benchmarks are fragmented, each focusing on a single capability such as image QA, robotic manipulation, or game playing. To measure true “generalist” intelligence—i.e., the ability to perceive, reason, and act across heterogeneous real‑world domains—the authors introduce MultiNet v1.0, a unified benchmark that aggregates six foundational capability regimes: visual grounding, spatial reasoning, tool use, physical commonsense, multi‑agent coordination, and continuous robot control.

MultiNet v1.0 integrates six heterogeneous datasets: ODINW for in‑the‑wild object detection, SQA3D for 3‑D egocentric spatial QA, PIQA for physical commonsense, Overcooked‑AI for cooperative multi‑agent gameplay, Open‑X Embodiment for continuous robotic manipulation, and BFCL v3 for multi‑turn function‑calling workflows. Each sub‑benchmark demands a distinct input‑output modality (image, video, text) and a specific output format (text, discrete action token, continuous action vector, API call). The authors provide an open‑source SDK that standardizes data loading, prompt construction, and result parsing, together with a reproducible submission pipeline that enforces zero‑shot evaluation across all tasks.

Three state‑of‑the‑art models are evaluated: GPT‑5 (a large multimodal LLM with strong vision‑language capabilities but no explicit action head), π0 (a flow‑matching VLA trained on ~903 M timesteps of robotic data), and Magma (a multi‑task foundation model that jointly learns vision, language, and action). Evaluation metrics are domain‑specific: Exact Match Rate for discrete actions, Mean Squared Error for continuous control, accuracy/F1 for text‑based QA, and success rate for API calls. Additionally, the authors introduce “output format stability” and “modality alignment” scores to capture whether a model’s predictions respect the required modality.

Results reveal a consistent pattern: all models excel within their training distribution but suffer severe degradation when faced with unseen domains or cross‑modal shifts. GPT‑5, despite being the most capable vision‑language model, achieves only ~27 % EMR on Overcooked and ~31 % accuracy on SQA3D, and its function‑calling success drops to 22 %. π0 retains impressive continuous control performance (MSE ≈ 0.04) but virtually loses language generation, often emitting continuous‑action tokens in response to textual prompts. Magma exhibits pervasive output‑modality confusion, producing mismatched formats in roughly 45 % of cases across the suite.

The authors attribute these failures to three root causes: (1) insufficient modality alignment—internal representations for vision, language, and action are not jointly calibrated, leading to representation collapse under transfer; (2) output‑format instability—current architectures lack a dynamic decoder that can switch between text, discrete, and continuous token streams; and (3) domain‑transfer fragility—training data are heavily biased toward specific visual conditions or robot morphologies, limiting generalization to novel physics or visual variations.

Beyond the empirical findings, the paper contributes a fully open‑source benchmark suite, a standardized evaluation protocol, and novel diagnostic metrics that go beyond raw accuracy. By making the code, data, and leaderboards publicly available, the authors enable reproducible research and provide a concrete target for future model development.

In conclusion, MultiNet v1.0 demonstrates that today’s leading multimodal and embodied models are still far from achieving true generalist intelligence. Progress will require research on shared multimodal representation spaces, adaptive output decoders, and more diverse, real‑world‑centric pre‑training that mitigates domain bias. The benchmark sets a clear, rigorous yardstick for measuring such advances.