用浏览器游戏评估代码代理的真实交付能力,看它能不能做出可玩的完整应用。
WebGameBench: Requirement-to-Application Evaluation for Coding Agents via Browser-Native Games

- 通过真实浏览器运行游戏来测试代码代理生成的应用
- 12个代理中最高可用率为76.9%,优秀率仅20.2%
- 首个基于人类可玩性验证的端到端游戏交付评测基准
代码代理越来越多被用于构建应用,但多数评估仍停留在源码、仓库级测试或中间过程,而非最终交付的应用。我们提出WebGameBench,一个从结构化游戏需求到可浏览器访问游戏的端到端评测基准。浏览器原生游戏提供紧凑但行为密集的测试环境:即使简单游戏也需协调输入处理、空间映射、规则执行、状态转换、终止条件、重启机制和可视反馈。在WebGameBench中,每个生成物均按统一部署协议构建、服务并暴露为浏览器可访问应用。运行时评估器在真实浏览器中与交付游戏交互,并打三类标签:EXCELLENT(优秀)、USABLE(可用)、UNUSABLE(不可用)。在人工评审子集上,运行时标签与人类游戏体验评审在可用率标准下高度一致。在111个任务、12个代码代理和14种评估配置下,最佳配置达到76.9%可用率,但仅20.2%优秀率。这一差距表明,达到最低可玩门槛远未实现完整需求满足。据我们所知,WebGameBench是首个将交付应用运行时标签与独立人类游戏评审在可用率标准下对齐的浏览器原生游戏交付评测基准。
原文摘要 · Abstract (English)
Coding agents are increasingly used as application builders, yet many evaluations still focus on source code, repository-level tests, or intermediate traces rather than the delivered application. We introduce WebGameBench, a requirement-to-application benchmark that evaluates whether coding agents can turn a frozen Structured WebGame Specification into a browser-accessible game. Browser-native games provide a compact but behavior-dense testbed: even simple games require coordinated input handling, spatial mapping, rule execution, state transitions, terminal conditions, restart behavior, and visible feedback. In WebGameBench, each generated artifact is built, served, and exposed as a browser-accessible application under a unified deployment protocol. A runtime evaluator then interacts with the delivered game in a real browser and assigns a three-way label: EXCELLENT, USABLE, or UNUSABLE. On a human-reviewed subset, the runtime label is broadly aligned with human gameplay review under the Usable-rate criterion. Across 111 tasks, 12 coding agents, and 14 evaluation configurations, WebGameBench separates current systems: the best configuration reaches a 76.9% usable rate but only a 20.2% excellent rate. This gap shows that crossing the minimum playable-delivery threshold is still far from complete requirement satisfaction. To our knowledge, WebGameBench is the first requirement-to-application benchmark for browser-native game delivery that validates delivered-application runtime labels against independent human gameplay review under the Usable-rate criterion.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。