用弱模型检测代码通过公开测试后仍存在的隐藏缺陷
Code Monitor Red Teaming for Public-Test-Passing Code
- 设计红队测试协议，在公开测试边界下评估弱验证器发现隐性错误的能力
- 71000个生成代码中43677个通过公开测试，其中23081个在隐性测试中失败
- 适合关注大模型代码鲁棒性与验证机制的开发者和研究者
公开测试是大语言模型生成代码的常见准入门槛，但通过测试并不意味着符合规范。我们研究一种部署场景下的监控问题：代码通过公开测试后，能否由更弱的LLM验证器识别残留的隐藏缺陷？提出代码监控红队测试框架，固定公开测试信息边界，同时变化生成压力、验证器结构和从弱到强的能力。构建CodeMonitorBench基准，覆盖函数级、数据科学和工作流代码。在71,000个生成样本中，43,677个通过公开测试，其中23,081个在隐性测试中失败。弱验证器在结构支持和模型家族提升下表现改善，但在5%误报率下仍遗漏多数隐藏缺陷。作为鲁棒性压力测试，对抗性公开测试过拟合使验证器的AUROC下降，低误报率下的漏检率上升。一个GLM-5.1验证器在相同证据边界下部分弥补差距；可解释性审计显示，剩余漏检源于验证器能力不足与M1证据限制的混合。
原文摘要 · Abstract (English)
Visible tests are a common gate for LLM-generated code, but passing them does not certify specification correctness. We study a deployment-like monitoring problem: after code has passed public tests, can a weaker LLM verifier identify the residual hidden bugs? We introduce Code Monitor Red Teaming, a monitor-red-teaming protocol that fixes a public-check information boundary while varying generator pressure, verifier scaffolding, and weak-to-strong capability. We instantiate it as CodeMonitorBench, spanning function-level, data-science, and workflow code. Across 71,000 generated candidates, 43,677 pass public tests and 23,081 of those fail hidden tests. Weak verifiers improve with scaffolding and model family, but still miss most hidden bugs at 5% false-positive rate. As a robustness stress test, adversarial public-test-overfit pressure lowers verifier AUROC and raises low-FPR miss rates in most cells. A GLM-5.1 verifier recovers part of the gap under the same evidence boundary; an inferability audit shows that remaining misses mix verifier failures with M1 evidence limits.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。