arXiv:2605.02909cs.LGcs.AI2026-05被引 2

系统性验证错误会严重破坏强化学习推理,导致模型性能停滞甚至崩溃。

Delay, Plateau, or Collapse: Evaluating the Impact of Systematic Verification Error on RLVR

论文配图:Delay, Plateau, or Collapse: Evaluating the Impact of Systematic Verification Error on RLVR
图 1 · 摘自论文原文
  • 分析系统性验证错误对奖励信号的影响,发现其模式比错误率更关键。
  • 虚假阳性会导致模型陷入次优状态或完全失效,而虚假阴性影响较小。
  • 提醒研究者:验证器质量不能只看平均错误率,需关注错误分布模式。

基于可验证奖励的强化学习(RLVR)已成为提升大语言模型推理能力的重要方法。尽管RLVR适用于具有可验证真实答案的任务,但实际验证器(如静态代码检查工具)可能引入错误的奖励信号。以往分析通常将此类错误视为独立随机噪声,认为仅延缓训练而不影响最终性能。然而,现实中的验证器往往存在系统性错误,可能导致模型从结构错误的奖励信号中学习到不良的稳定行为。本文在算术任务上通过受控实验研究了系统性验证错误的影响。结果表明,系统性假阴性与随机噪声效果类似;而系统性假阳性则引发从次优平台到性能崩溃的多种行为。关键在于,这些后果不取决于整体错误率,而由错误的具体模式决定,使得事前缓解难以实现。研究揭示,相较于以往结论,现实验证错误会显著塑造RLVR结果,验证器质量必须超越样本层面的错误率来理解。

原文摘要 · Abstract (English)

Reinforcement Learning with Verifiable Rewards (RLVR) has become a powerful approach for improving the reasoning capabilities of large language models (LLMs). While RLVR is designed for tasks with verifiable ground-truth answers, real-world verifiers (e.g., static code checkers) can introduce errors into the reward signal. Prior analyses have largely treated such errors as random and independent across samples, concluding that errors merely slow training with limited effect on final performance. However, practical verifiers tend to exhibit systematic errors. This introduces a risk of models learning unwanted consistent behavior from a structurally incorrect reward signal. In this work, we study the impact of such systematic verification errors on RLVR. Through controlled experiments on arithmetic tasks, we show that systematic false negatives lead to similar effects as random noise. On the other hand, systematic false positives can cause a wide range of behaviors from sub-optimal plateaus to performance collapse. Crucially, these outcomes are not determined by the overall error rate but by the specific pattern of introduced errors, making pre-hoc mitigation difficult. Our results show that, in contrast to prior conclusions, realistic verification errors can critically shape RLVR outcomes and that verifier quality has to be understood beyond its sample-level error rate.

强化学习大模型验证误差推理优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。