首个游戏开发评测基准，测试智能体在复杂多模态任务中的表现
GameDevBench: Evaluating Agentic Capabilities Through Game Development
- 构建333个来自教程的游戏开发任务，需处理代码与图像动画等多模态数据
- 最佳模型仅解决53.8%任务，2D图形类任务成功率低至33.0%
- 引入图像视频反馈机制，显著提升性能，最高达52.0%
尽管编码智能体发展迅速，其多模态版本进展却滞后。核心挑战在于缺乏融合软件开发复杂性与深度多模态理解的评测环境。游戏开发中，智能体需在庞大代码库中操作着色器、精灵、动画等内在多模态资源。我们提出GameDevBench，首个面向游戏开发任务的智能体评测基准。该基准包含333项任务，源自网络与视频教程，要求高度多模态理解，平均解决方案代码量和文件修改量超此前基准三倍以上。现有智能体表现不佳，最优模型仅完成53.8%任务。任务难度与多模态复杂度强相关：以游戏玩法为主的任务成功率为51.4%，而2D图形任务仅为33.0%。为提升多模态能力，我们引入两种基于图像和视频的反馈机制，虽简单但效果显著，使GPT-5.4在视觉反馈下性能从41.1%提升至52.0%。
原文摘要 · Abstract (English)
Despite rapid progress on coding agents, progress on their multimodal counterparts has lagged behind. A key challenge is the scarcity of evaluation testbeds that combine the complexity of software development with the need for deep multimodal understanding. In game development, agents must navigate large, dense codebases while manipulating intrinsically multimodal assets such as shaders, sprites, and animations within a visual game scene. We present GameDevBench, the first benchmark for evaluating agents on game development tasks. GameDevBench consists of 333 tasks derived from web and video tutorials. Tasks require significant multimodal understanding and are complex: the average solution requires over three times the lines of code and file changes compared to prior software development benchmarks. Agents struggle with game development, with the best agent and method solving only 53.8% of tasks. We find a strong correlation between perceived task difficulty and multimodal complexity, with average success rate dropping from 51.4% on gameplay-oriented tasks to 33.0% on 2D graphics tasks. To improve multimodal capability, we introduce two simple image- and video-based feedback mechanisms for agents. Despite their simplicity, these methods consistently improve performance, increasing GPT-5.4's performance from 41.1% to 52.0% when given visual feedback.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。