评测编码代理在游戏开发全流程中的表现，发现其造出可玩原型强但查错优化弱。
GameXpert-Bench: How Far Are Coding Agents from Expert Game Development?
- 分生成、修复、优化三阶段构建评估体系
- 97个游戏生成任务、100个含19-27个漏洞的修复任务
- 适合关注智能编程与游戏自动化研发的人
近期大语言模型可作为编码代理，从自然语言请求中构建完整游戏。游戏开发要求程序逻辑、视觉音频、界面交互与可玩性协同工作于单一可执行产物中，因此需同时评估游戏成品与开发过程。现有基准多仅评估最终产物或孤立开发阶段。我们分析人类-代理完整开发轨迹，识别出三个贯穿游戏开发生命周期的阶段：初始生成、缺陷诊断与修复、多轮优化。为此提出 GameXpert-Bench，将三阶段转化为三个互补的评测赛道：GameGen 评估单次请求下从空环境生成完整游戏的能力；GameFix 评估在缺陷被报告或由代理自行发现时的诊断与修复能力；GameOpt 评估基于真实用户-代理开发轨迹的请求链式累积优化。每个赛道采用实时游戏交互、确定性行为测试或最终产品标准结合回归检查进行评估。该套件包含 97 个生成任务（覆盖 11 种类型）、100 个修复任务（来自 50 个游戏关卡，每关注入 19-27 个漏洞）及 17 条优化链（每条 6 轮，共 102 次请求）。结果表明，当前代理在生成可玩基础版本和实现显式需求方面更可靠，但在发现缺陷、验证运行行为以及保持变更后功能完整性方面仍显著不足。
原文摘要 · Abstract (English)
Recent large language models (LLMs) can operate as coding agents that build complete games from natural language requests. Game development is especially demanding because program logic, visual and audio content, interfaces, interaction and playability must function together in one executable artifact. Measuring this capability therefore requires evaluation of both game product and the development process. Existing benchmarks often assess the game development capabilities of LLMs by evaluating the final artifact or an isolated development stage. Our analysis of complete human-agent development trajectories identifies three stages that together span the lifecycle of game development with a coding agent: initial game generation, bug diagnosis and repair, and optimization over multiple turns. Therefore, we introduce GameXpert-Bench, which operationalizes the three lifecycle stages as three complementary benchmark tracks. GameGen evaluates complete game creation from a single request in an empty workspace. GameFix evaluates diagnosis and repair when defects are reported or left for the agent to discover. GameOpt evaluates cumulative optimization through request chains seeded by real development trajectories between users and agents. We evaluate each track using live game interaction, deterministic behavioral tests, or final product criteria with regression checks. The suite contains 97 generation tasks across 11 genres; 100 repair tasks from 50 game levels verified by humans, each with 19-27 injected bugs; and 17 optimization chains with six turns and 102 requests. Across the three tracks, current agents are more reliable at producing playable foundations and implementing explicit requirements than at discovering defects, verifying runtime behavior, and preserving functionality across changes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。