在真实游戏引擎中测试代码智能体，发现其在复杂C++开发中仍表现不佳。
GameEngineBench: Evaluating Coding Agents on Real C++ Runtime Environments
- 构建了基于Unreal Engine 5的真实项目基准，评估代码智能体修改C++的能力。
- 最强模型仅55.5%任务通过，31个任务全未解决。
- 适合研究实时系统编程与智能体协作的开发者参考。
游戏引擎提供实时仿真、渲染、物理、交互、网络和资源管线，不仅用于游戏，还在医疗、机器人、建筑、制造等领域具有重要应用。由于游戏开发是这些系统最成熟且公开可用的领域，因此成为评估需在状态化、交互式实时系统中修改C++代码的代码智能体的理想测试平台。本文提出GameEngineBench，一个基于九个真实游戏仓库构建的基准，用于评估代码智能体在Unreal Engine 5项目中的特定C++实现任务。评估集包含110个任务，涵盖玩法机制、多人行为、AI与世界调度、动画与移动、UI与会话代码、加载行为、在线服务集成、持久化、数据序列化、XR行为及渲染插件等。这些任务要求模型做出可编译且通过行为测试的原生C++修改。在十二种配置中，最强模型达到55.5% pass@1，仍有31个任务被所有配置未能解决。结果表明，前沿代码智能体在实时交互软件的深度集成C++开发中仍面临挑战，凸显游戏引擎基准对现有软件工程评估的重要补充价值。
原文摘要 · Abstract (English)
Game engines provide real-time simulation, rendering, physics, interaction, networking, and asset pipelines, making them valuable not only for games but also for 3D applications in healthcare, robotics, architecture, manufacturing, and related domains. Because game development is where these systems are most mature and publicly available, it offers a practical testbed for evaluating coding agents that must modify C++ code within stateful, interactive, real-time systems. We present GameEngineBench, a benchmark for evaluating coding agents on scoped C++ implementation tasks inside Unreal Engine 5 projects, built from nine real-world game repositories. The evaluation set consists of 110 tasks spanning gameplay mechanics, multiplayer behavior, AI and world orchestration, animation and movement, UI and session code, loading behavior, online-service integration, persistence, data serialization, XR behavior, and rendering-oriented plugins. These tasks require models to make native C++ changes that compile and satisfy behavioral tests within executable Unreal Engine projects. Across twelve evaluated configurations, the strongest model reaches 55.5\% pass@1, while 31 tasks remain unsolved by every configuration. Our results demonstrate that frontier coding agents continue to struggle with deeply integrated C++ development for real-time interactive software, highlighting game-engine benchmarks as a valuable complement to existing software engineering evaluations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。