改进验证器减少误判，让大模型推理强化学习更有效
TinyV: Reducing False Negatives in Verification Improves RL for LLM Reasoning
- 用轻量级LLM动态识别并修复验证器的误判
- 在数学推理任务中提升通过率最高达10%，加速收敛
- 适合做强化学习微调的大模型研究者参考
强化学习（RL）通过奖励信号优化策略，已成为提升大语言模型（LLMs）推理能力的重要工具。然而，其效果依赖于奖励的可靠性，而奖励由验证器提供。本文揭示并分析了广泛存在的“假阴性”问题：验证器错误拒绝正确输出。对Big-Math-RL-Verified数据集的深入研究显示，超过38%的模型生成结果存在假阴性，验证器未能识别正确答案。我们通过实证与理论证明，假阴性会剥夺模型有效的梯度信号，严重阻碍训练并延缓收敛。为此，提出tinyV——一种轻量级基于LLM的验证器，可增强现有基于规则的方法，动态识别潜在假阴性并恢复有效响应，从而获得更准确的奖励估计。在多个数学推理基准上，集成tinyV后通过率最高提升10%，且收敛速度优于基线。研究强调解决验证器假阴性的重要性，并提供实用方案以改进基于RL的LLM微调。
原文摘要 · Abstract (English)
Reinforcement Learning (RL) has become a powerful tool for enhancing the reasoning abilities of large language models (LLMs) by optimizing their policies with reward signals. Yet, RL's success relies on the reliability of rewards, which are provided by verifiers. In this paper, we expose and analyze a widespread problem--false negatives--where verifiers wrongly reject correct model outputs. Our in-depth study of the Big-Math-RL-Verified dataset reveals that over 38% of model-generated responses suffer from false negatives, where the verifier fails to recognize correct answers. We show, both empirically and theoretically, that these false negatives severely impair RL training by depriving the model of informative gradient signals and slowing convergence. To mitigate this, we propose tinyV, a lightweight LLM-based verifier that augments existing rule-based methods, which dynamically identifies potential false negatives and recovers valid responses to produce more accurate reward estimates. Across multiple math-reasoning benchmarks, integrating TinyV boosts pass rates by up to 10% and accelerates convergence relative to the baseline. Our findings highlight the critical importance of addressing verifier false negatives and offer a practical approach to improve RL-based fine-tuning of LLMs. Our code is available at https://github.com/uw-nsl/TinyV.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。