Virtual Community: An Open World for Humans, Robots, and Society

The rapid progress in AI and Robotics may lead to a profound societal transformation, as humans and robots begin to coexist within shared communities, introducing both opportunities and challenges. To explore this future, we present Virtual Community-an open-world platform for humans, robots, and society-built on a universal physics engine and grounded in real-world 3D scenes. With Virtual Community, we aim to enable the study of embodied social intelligence at scale. To support these, Virtual Community features: 1) An open-source multi-agent physics simulator that supports robots, humans, and their interactions within a society; 2) A large-scale, real-world aligned community generation pipeline, including vast outdoor space, diverse indoor scenes, and a community of grounded agents with rich characters and appearances. Leveraging Virtual Community, we propose two novel challenges. The Community Planning Challenge evaluates multi-agent reasoning and planning ability in open-world settings, such as cooperating to help agents with daily activities and efficiently connecting other agents. The Community Robot Challenge requires multiple heterogeneous robots to collaborate in solving complex open-world tasks. We evaluate various baselines on these tasks and demonstrate the challenges in both high-level open-world task planning and low-level cooperation controls. We hope that Virtual Community will unlock further study of human-robot coexistence within open-world environments.

💡 Research Summary

The paper introduces “Virtual Community,” an open‑world simulation platform designed to study embodied social intelligence at scale by bringing together human‑like avatars, heterogeneous robots, and realistic environments grounded in real‑world 3D data. The system consists of two major components. First, a physics‑based multi‑agent simulator built on the Genesis engine provides a unified framework where both humanoid agents and various robot types (wheeled, mobile manipulators, etc.) can interact under consistent physical laws, sharing observation and action spaces while supporting detailed collision, force, and friction modeling. Second, an automated pipeline converts publicly available geospatial data (Google Maps, OpenStreetMap) into large‑scale indoor‑outdoor scenes. The pipeline performs mesh simplification, texture in‑painting using Stable Diffusion 3, and object insertion via generative models (One‑2‑3‑45) to produce clean geometry and high‑fidelity textures suitable for ground‑level rendering and physics simulation.

On top of these environments, the authors employ GPT‑4o to generate rich character profiles for human agents, including age, occupation, values, financial status, group affiliations, and personal goals. A social relationship network is also automatically constructed, encoding affinities and potential collaborations among agents. This “society layer” enables the definition of complex social tasks that go beyond simple navigation or object manipulation.

Two novel challenges are built on the platform. The Community Planning Challenge focuses on high‑level multi‑agent reasoning and planning in open worlds. Sub‑tasks include “assistant tasks” where agents must help others with daily activities (e.g., providing transport, delivering items) and “social influence tasks” where agents must efficiently explore the community and establish connections. The challenge highlights the scalability problems of current multi‑agent reinforcement learning (MARL) methods, as state‑action spaces explode with the number of agents and the dynamic nature of the environment. The Community Robot Challenge tests heterogeneous robot teams that must cooperate across indoor and outdoor spaces to accomplish composite tasks, requiring tight integration of low‑level control (e.g., grasping, locomotion) with high‑level coordination and interaction with human agents.

The authors evaluate several baselines: classic planning algorithms, graph‑neural‑network‑based policies, and MARL approaches. Results show that while some methods can achieve moderate success on high‑level objectives (≈70 % success in reaching goals or forming connections), they struggle with fine‑grained robot cooperation, exhibiting error rates above 40 % in tasks such as object hand‑offs or synchronized navigation. These findings expose a gap between existing algorithms and the demands of realistic, physics‑rich, socially complex open worlds.

All code, data, and the simulation environment are released as open source, allowing the research community to reproduce experiments, extend the world generation pipeline, and design new tasks. By providing a scalable, realistic testbed that unifies human and robot agents within geospatially grounded environments, Virtual Community opens avenues for research on embodied general intelligence, large‑scale multi‑agent coordination, and the practical deployment of robot teams in human societies. The work positions itself as a foundational platform for the next generation of human‑robot coexistence studies.