GameDevBench: Evaluating Agentic Capabilities Through Game Development

Despite rapid progress on coding agents, progress on their multimodal counterparts has lagged behind. A key challenge is the scarcity of evaluation testbeds that combine the complexity of software development with the need for deep multimodal understanding. Game development provides such a testbed as agents must navigate large, dense codebases while manipulating intrinsically multimodal assets such as shaders, sprites, and animations within a visual game scene. We present GameDevBench, the first benchmark for evaluating agents on game development tasks. GameDevBench consists of 132 tasks derived from web and video tutorials. Tasks require significant multimodal understanding and are complex – the average solution requires over three times the amount of lines of code and file changes compared to prior software development benchmarks. Agents still struggle with game development, with the best agent solving only 54.5% of tasks. We find a strong correlation between perceived task difficulty and multimodal complexity, with success rates dropping from 46.9% on gameplay-oriented tasks to 31.6% on 2D graphics tasks. To improve multimodal capability, we introduce two simple image and video-based feedback mechanisms for agents. Despite their simplicity, these methods consistently improve performance, with the largest change being an increase in Claude Sonnet 4.5’s performance from 33.3% to 47.7%. We release GameDevBench publicly to support further research into agentic game development.

💡 Research Summary

The paper introduces GameDevBench, the first benchmark specifically designed to evaluate multimodal capabilities of coding agents through game development tasks. Leveraging the open‑source Godot 4 engine, the authors collected 57 video tutorials from YouTube and 31 web tutorials, each paired with a permissively‑licensed GitHub repository. After automatic transcription, repository matching, and a multi‑stage pipeline (automatic task generation, refinement, and human annotation), they produced 132 base tasks and 17 variants. Each task requires modifications across an average of five files and 106 lines of code, more than three times the size of solutions in existing software‑development benchmarks such as SWE‑Bench. Moreover, 82 % of tasks involve additional assets (PNG images, TTF fonts, GDShader files, WAV audio, etc.), ensuring a strong multimodal component.

Tasks are categorized along two axes: skill type (Gameplay Logic, 3D Graphics & Animation, 2D Graphics & Animation, User Interface) and editor type (Scene, Script, and contextual editors such as Animation, Audio, Shader, TileSet). Distribution is roughly 36 % Gameplay, 26 % 3D graphics, 20 % 2D graphics, and 16 % UI. Evaluation uses Godot’s built‑in unit‑testing framework, allowing deterministic verification of game behavior, physics interactions, and visual outcomes without relying on subjective visual judges.

Baseline experiments with state‑of‑the‑art LLM‑based agents (Claude Sonnet 4.5, GPT‑4, GPT‑3.5, etc.) show limited success: the best model solves only 54.5 % of tasks, with a marked drop on 2D graphics tasks (31.6 %). To improve multimodal understanding, the authors introduce two lightweight feedback mechanisms: (1) a Model Context Protocol (MCP) server that streams real‑time screenshots of the editor, and (2) a video feed capturing the running game scene. Both methods consistently boost performance; for example, Claude Sonnet 4.5 rises from 33.3 % to 47.7 % success. These results demonstrate that providing agents with direct visual or video context can substantially enhance their ability to reason about and manipulate multimodal assets.

The paper also details the full data‑pipeline, including prompts, checklists, and annotation guidelines, and releases all code, assets, and benchmark tasks publicly. Future directions suggested include extending the benchmark to other engines (Unity, Unreal), exploring human‑agent collaborative workflows, and developing richer multimodal LLM architectures. Overall, GameDevBench offers a rigorous, reproducible platform for measuring and advancing the multimodal reasoning and coding abilities of next‑generation AI agents.