验证器不完美也能有效训练大模型，15%错误率下性能仅降2个百分点。
An Imperfect Verifier is Good Enough: Learning with Noisy Rewards
- 在强化学习中引入噪声验证，测试模型对错误反馈的鲁棒性。
- 15%错误率下验证准确率仍比干净基线低不超过2个百分点。
- 建议优先高精度而非完美验证，适合实际部署中的模型训练。
基于可验证奖励的强化学习（RLVR）已成为大语言模型后训练的重要方法。然而，验证器极少完全无误；即使确定性检查也可能出错，而日益依赖模型化评判者更加剧了这一问题。当前尚不清楚RLVR对噪声的容忍度及所需验证准确率。我们通过在代码生成与科学推理任务中引入噪声，系统研究了这一问题。结果表明，噪声率高达15%时，验证准确率仍比无噪声基线低不超过2个百分点。该结论在控制型与模型型噪声、三种模型家族（Qwen3、GLM4、Llama 3.1）及4B至9B不同规模下均一致成立。整体表明，不完美的验证并非RLVR的根本障碍。此外，研究提示实践者应优先选择中等准确率但高精度的验证方式。
原文摘要 · Abstract (English)
Reinforcement Learning with Verifiable Rewards (RLVR) has become a prominent method for post-training Large Language Models (LLMs). However, verifiers are rarely error-free; even deterministic checks can be inaccurate, and the growing dependence on model-based judges exacerbates the issue. The extent to which RLVR is robust to such noise and the verifier accuracy required for effective training remain unresolved questions. We investigate these questions in the domains of code generation and scientific reasoning by introducing noise into RL training. Noise rates up to 15% yield peak validation accuracy within 2 percentage points of the clean baseline. These findings are consistent across controlled and model-based noise types, three model families (Qwen3, GLM4, Llama 3.1), and model sizes from 4B to 9B. Overall, the results indicate that imperfect verification does not constitute a fundamental barrier to RLVR. Furthermore, our findings suggest that practitioners should prioritize moderate accuracy with high precision over perfect verification.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。