发现代码强化学习环境易被错误解骗，提出用金标准检测加固方案。
Auditing Reward Hackability in Code RL Training Environments
- 用生成测试用例方式探测环境漏洞，发现28.5%任务可被错误补丁通过。
- 在134个模型中，漏洞任务上通过率比稳健任务高14.14个百分点。
- 引入金标准校验环，使9/11个漏洞任务成功升级为抗欺骗环境。
我们测量代码强化学习环境中接受错误解的频率。在SWE-bench Verified的49个任务样本中，28.5%的任务测试套件过于薄弱，导致一个经Docker验证的错误补丁能通过测试。在6个仓库的20个R2E-Gym任务中，单次生成攻击的成功率为25.0%。对SWE-bench Verified中134个前沿模型提交结果进行随机效应元分析发现，在同一人类评级难度层级下，漏洞任务上的模型Pass@1比稳健任务高出14.14个百分点（95%置信区间[+11.80, +16.48]；单侧p < 10^-6；I² = 0%；123/134个模型呈正向差异）。随后我们提出一种加固方法：在调用LLM评测前，先用内联的LLM裁判结合Docker金标准校验每个生成的测试用例是否在真实解上失败。在审计的11个漏洞任务中，该校验环节识别出105个关键测试中有65个在真实补丁上也失败，表明原始的LLM裁判漏掉了61.9%的缺陷。通过多样性偏向重试机制，该循环将9个任务成功升级为带防护的版本。
原文摘要 · Abstract (English)
We measure the rate at which code RL environments accept incorrect solutions as correct. On a 49-task sample of SWE-bench Verified, 28.5% of tasks have test suites weak enough that a Docker-verified incorrect patch passes them. On 20 R2E-Gym tasks across 6 repositories, the same pipeline at single-shot exploit generation yields 25.0%. A random-effects meta-analysis over 134 frontier model submissions to SWE-bench Verified finds, within the same human-rated difficulty stratum, model Pass@1 is +14.14 percentage points higher on flagged-hackable tasks than on robust ones (95% CI [+11.80, +16.48]; one-sided p < 10^-6; I^2 = 0%; 123 of 134 models positive). We then describe a procedure for hardening the broken tasks. An inline LLM judge with a Docker gold-sanity gate runs each generated test against the gold solution before the judge is consulted. On the 11 broken tasks in the audit, the gate flags 65 of 105 decisive LLM-generated tests as failing on the gold patch itself, a 61.9% per-augmentation defect rate the LLM judge alone misses. With diversity-biased retry, the loop converges 9 of 11 tasks to a gated upgrade.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。