arXiv:2607.11022cs.LGcs.CL2026-07被引 1

发现代码生成奖励测试存在顽固错误,会持续误判错误代码为正确。

When the Reward Suite Is Leaky: A Preregistered Causal Contrast of Natural Verifier False Positives in RLVR

论文配图:When the Reward Suite Is Leaky: A Preregistered Causal Contrast of Natural Verifier False Positives in RLVR
图 1 · 摘自论文原文
  • 通过对比原始测试与加固测试,验证奖励漏洞导致模型持续误判。
  • 误判率高出干净任务43.8个百分点,且可提前用静态审计检测。
  • 误判主要源于预存错误模式,非模型学习所致,适合关注RLVR安全性的研究者。

用于代码生成的强化学习验证奖励(RLVR)测试套件存在自然假阳性:每项任务中持久、不对称的错误会反复接受相同的错误程序,不同于现有分析所假设的对称或重采样噪声。我们在已部署的套件上开展预注册的双臂因果对比实验:在相同MBPP任务、种子和计算资源下,使用原始MBPP测试(有漏洞)与MBPP+额外测试(加固)作为奖励,训练GRPO模型。两次独立家族复现该设计,且预注册在数据产生前冻结。结果表明:平均保留效果受限,未低于预注册1.5分阈值(差距0.20分,单侧95%上限0.75分)。受奖励驱动的假阳性比例与训练前的静态泄漏审计高度相关(Spearman 0.80),且注册训练端测试显示,泄漏分组的假阳性占比比干净任务高出43.8个百分点。经人工审核确认,有47.57%的受奖励假阳性真实为错误代码,两个复现家族均再现高比例。机制证据支持预存错误模式选择,而非模型学习利用:假阳性在实验周期内未增长,未训练的基础模型在漏斗过滤下即产生相同错误输出。进一步将同一工具应用于前沿评判者自身:其对自身假阳性的自评偏弱,同作者测试无结论,甚至最高评分读者在弱策略错误上的得分仍远低于自身表现——仅在两个MBPP任务上测试,无法推断前沿模型普遍性。

原文摘要 · Abstract (English)

The test suites used as RLVR rewards for code have natural false positives: per-task, persistent, asymmetric errors that accept the same wrong programs every time they appear, unlike the symmetric or resampled noise assumed by existing noise-robustness analyses. We run a preregistered two-arm causal contrast on a deployed suite: GRPO on identical MBPP tasks, seeds, and compute, rewarded by the original MBPP tests (leaky) versus the MBPP+ extra tests (hardened). Two further families replicate the design under a preregistration frozen before their data existed. [C] The average held-out effect is bounded: non-inferior under a preregistered 1.5-pt margin (gap 0.20 pt, one-sided 95% upper bound 0.75 pt). [C] Rewarded false-positive mass tracks a cheap static leakiness audit computed before training (Spearman 0.80), and the registered train-side test puts the leak-stratum FP share +43.8 pt above clean tasks. [E] Auditing every rewarded FP under signed, human-adjudicated rules finds a large residual of verified genuinely wrong code: 47.57% record-weighted; both replication families reproduce a large share. The reward paid for real bugs, not merely suite artifacts. [E] Mechanism evidence is consistent with selection of pre-existing error modes rather than learned exploitation: FP incidence does not grow within our horizon, and untrained base models already produce the same wrong outputs under the leaky filter. We then turn the same instrument on the frontier judges themselves: on their own false positives they self-assess only weakly, a same-author test is unresolved, and even the highest-scoring reader we probe stays far below its score on a weaker policy's errors -- two subjects on MBPP, licensing nothing about frontier models in general. A cheap static audit locates exposure before training; hardening the reward removes the measurement inflation, though here it buys little capability.

强化学习代码生成奖励漏洞假阳性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。