PLAICraft: Large-Scale Time-Aligned Vision-Speech-Action Dataset for Embodied AI

Advances in deep generative modeling have made it increasingly plausible to train human-level embodied agents. Yet progress has been limited by the absence of large-scale, real-time, multi-modal, and socially interactive datasets that reflect the sensory-motor complexity of natural environments. To address this, we present PLAICraft, a novel data collection platform and dataset capturing multiplayer Minecraft interactions across five time-aligned modalities: video, game output audio, microphone input audio, mouse, and keyboard actions. Each modality is logged with millisecond time precision, enabling the study of synchronous, embodied behaviour in a rich, open-ended world. The dataset comprises over 10,000 hours of gameplay from more than 10,000 global participants. Alongside the dataset, we provide an evaluation suite for benchmarking model capabilities in object recognition, spatial awareness, language grounding, and long-term memory. PLAICraft opens a path toward training and evaluating agents that act fluently and purposefully in real time, paving the way for truly embodied artificial intelligence.

💡 Research Summary

PLAICraft introduces a novel, large‑scale dataset and collection platform designed to advance research in embodied artificial intelligence (EAI). The authors argue that existing Minecraft‑based datasets such as MineRL, VPT, CraftAssist, and MineDojo suffer from limitations including single‑player focus, asynchronous interaction, task‑centric design, and lack of real‑time multimodal alignment. To overcome these constraints, PLAICraft records multiplayer Minecraft sessions across five modalities—screen video, game output audio, microphone input audio, mouse actions, and keyboard actions—with millisecond‑precision timestamps, thereby preserving the causal relationship between inputs and their perceptual consequences.

The dataset comprises over 10,000 hours of gameplay from more than 10,000 globally distributed participants. Each session lasts up to four hours and is captured in a persistent, open‑world environment that evolves continuously without episodic resets. Players interact through a proximity‑based voice chat plugin, enabling natural spoken dialogue whose audio streams are recorded alongside the game’s output sounds. This design captures rich social dynamics such as greetings, collaborative building, conflict, and coordinated combat, providing a realistic testbed for language grounding, social reasoning, and cooperative behavior.

Technically, the data collection infrastructure is built on Amazon Web Services. Users access the system via a web browser, authenticate with email or phone, and are routed to a dedicated EC2 g5.2xlarge instance equipped with an NVIDIA A10G GPU. The Minecraft client runs in full‑screen mode, streamed to the user through NICE DCV at up to 60 FPS and 2K resolution. Background recording processes capture each modality, store the raw streams in an S3 bucket, and log session metadata in DynamoDB. Lambda functions and API Gateway orchestrate the workflow, automatically terminating idle instances and uploading data, thereby ensuring privacy and scalability.

Key characteristics of the dataset include: (1) millisecond‑level temporal alignment across heterogeneous sampling rates (video at 10 fps, audio at 75 Hz, input events at 50–100 Hz), enabling precise study of sensorimotor loops; (2) a continuously evolving world state where player‑built structures, mined blocks, inventories, and experience persist across sessions, supporting research on long‑term memory and world modeling; (3) extensive social interaction data captured via voice chat, with automatic transcription and LLM‑based labeling to extract intents, emotions, and collaborative actions; (4) a broad spectrum of player activities ranging from basic locomotion and block placement to advanced redstone circuitry, Ender Dragon battles, and even exploit‑driven behaviors.

To facilitate systematic evaluation, the authors provide an evaluation suite comprising four benchmark families: object recognition (visual and auditory), spatial awareness (mapping, navigation, and viewpoint estimation), language grounding (command following, dialogue response, and collaborative instruction), and long‑term memory (recalling previously built structures or items to achieve new goals). Each benchmark includes standardized metrics (e.g., accuracy, IoU, BLEU, recall) and baseline protocols, allowing direct comparison between models and against human performance.

In conclusion, PLAICraft represents the first publicly available dataset that simultaneously offers real‑time multimodal alignment, persistent world dynamics, and authentic social interaction at a scale sufficient for training modern deep generative and reinforcement learning agents. By releasing both the dataset and the underlying cloud‑based collection pipeline as open resources, the authors enable the community to extend the approach to other virtual environments or even physical robotic platforms, paving the way toward truly embodied AI systems capable of fluent, purposeful action in complex, socially situated settings.