用智能代码代理发现官方评测集遗漏的错误代码。
Coding Agents as Test-Suite Auditors: Finding What Official Suites Miss While Approaching What They Catch
- 用现成代码代理生成对抗性测试用例,暴露官方评测集漏洞。
- 在AtCoder中发现589个被接受但含错的提交,五代理共识别906个。
- 无官方评测时可自建有效测试集,且性能优于现有方法。
在线判题系统的评测结果及其构建的数据集和基准常被视为评估和训练代码大模型的金标准。然而先前审计已警示:官方评测集会接受存在缺陷的提交。这些审计仅停留在警告层面,未提供实际解决方案。本文提出两阶段补救:第一,使用现成的代码代理作为测试集审计者,既能生成对抗性测试集以揭示官方评测集遗漏的问题,也能在缺乏官方评测集时自动生成测试集;第二,通过认证链验证代理标记的提交是否真正存在缺陷——多个独立编写的通过解法在每组测试上输出一致,暴力解法解决分歧,问题级验证器确保失败输入合法。一个代理在AtCoder的20,375个已通过提交中发现了589个经验证的、虽通过但含错的提交;五代理联合检测出至少906个此类提交。五个代理在逻辑错误覆盖度上均与官方评测集保持在1.7个百分点以内。在无官方评测集的Codeforces赛后题目上,该方法在所有测试预算下均优于五个复现基线。有官方评测时,代理用于审计评测集完备性而非默认其正确;无官方评测时,代理生成的测试集捕获了最多含错提交,优于对比方法。
原文摘要 · Abstract (English)
Online-judge verdicts and the datasets and benchmarks built on them are treated as ground truth for evaluating and training large language models for code. Yet prior audits have sounded a warning: official suites accept buggy submissions. These audits, however, stop at the warning and offer no practical remedy. Our remedy has two parts: an off-the-shelf coding agent, serving as a test-suite auditor, both builds adversarial test suites to expose what official suites miss and supplies these suites where no official suite exists; a certification chain determines whether each agent-flagged submission is genuinely buggy without relying on the official judge: multiple independently written accepted solutions agree on the expected output for every test, brute-force solutions settle disagreements, and a per-problem validator certifies each failing input legal. One such agent identifies 589 verified accepted-but-buggy submissions among AtCoder's 20,375 audited accepted submissions; extending the same certification to all five agents yields a union floor of 906 such submissions. Five agents, scored separately, each stay within 1.7pp of official-suite coverage on logic bugs those suites catch. On post-cutoff Codeforces problems with no available official suites, the same test-building method leads all five reproduced baselines at every tested input budget. Where an official suite exists, the agent audits suite adequacy instead of assuming it; where none exists, agent suites catch the most buggy submissions among methods we reproduced and tested.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。