用游戏漏洞检测挑战LLM质量保证能力，发现最牛模型也只能发现近半数漏洞。
GBQA: A Game Benchmark for Evaluating LLMs as Quality Assurance Engineers
- 构建多智能体系统自动生成带漏洞的游戏，人工验证确保质量。
- 顶尖模型仅识别48.39%的已验证漏洞，体现自主找错仍极难。
- 适合研究大模型软件测试、自动化质量保障的学者与工程师。
自主发现软件漏洞仍是现代开发中的重大挑战。相比代码生成，动态运行环境的复杂性使大语言模型（LLMs）在漏洞探测上更加困难。本文以游戏开发为典型场景，提出游戏质量保证基准GBQA，包含30个游戏和124个经人工验证的漏洞，涵盖三个难度等级，用于评估LLMs是否能自主发现软件缺陷。该基准通过多智能体系统可扩展地生成游戏并注入漏洞，同时引入人类专家确保正确性。此外，我们提供一个基于多轮ReAct循环和记忆机制的交互式基线智能体，支持跨不同LLM对游戏环境进行长时程探索。在前沿LLM上的大量实验表明，自主漏洞发现仍极具挑战：表现最佳的Claude-4.6-Opus在思考模式下仅能发现48.39%的已验证漏洞。我们认为GBQA提供了合适的测试平台与评估标准，其进展将有助于缩小自主软件工程的差距。
原文摘要 · Abstract (English)
The autonomous discovery of bugs remains a significant challenge in modern software development. Compared to code generation, the complexity of dynamic runtime environments makes bug discovery considerably harder for large language models (LLMs). In this paper, we take game development as a representative domain and introduce the Game Benchmark for Quality Assurance (GBQA), a benchmark containing 30 games and 124 human-verified bugs across three difficulty levels, to evaluate whether LLMs can autonomously detect software bugs. The benchmark is constructed using a multi-agent system that develops games and injects bugs in a scalable manner, with human experts in the loop to ensure correctness. Moreover, we provide a baseline interactive agent equipped with a multi-round ReAct loop and a memory mechanism, enabling long-horizon exploration of game environments for bug detection across different LLMs. Extensive experiments on frontier LLMs demonstrate that autonomous bug discovery remains highly challenging: the best-performing model, Claude-4.6-Opus in thinking mode, identifies only 48.39% of the verified bugs. We believe GBQA provides an adequate testbed and evaluation criterion, and that further progress on it will help close the gap in autonomous software engineering.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。