让大模型推理更聪明：训练时加个验证器，测试时算得更快更准
Putting the Value Back in RL: Better Test-Time Scaling by Unifying LLM Reasoners With Verifiers
- 训练时让模型同时学推理和自验证，用强化学习数据联合优化
- 数学题准确率提升超20%，测试时并行计算效率提高8到32倍
- 适合需要高可靠推理的场景，尤其擅长处理难任务和陌生领域
当前主流的微调大模型推理能力的强化学习方法（如GRPO或留一法PPO）舍弃了已学的价值函数，转而依赖经验估算的回报。这限制了测试时计算资源的扩展能力，因为价值函数本可用于验证。然而，若部署时已规划并行计算，则训练应为此做准备。本文提出RL$^V$，通过联合训练大模型作为推理器与生成式验证器，仅需少量额外开销即赋予其验证能力。实证表明，使用并行采样时，RL$^V$使MATH基准准确率提升超过20%，相比基础强化学习方法实现8-32倍的高效测试时计算扩展。此外，该方法在从易到难及域外任务上均表现出强泛化能力；当同时扩展并行与串行计算资源时，采用长推理路径的R1模型性能提升1.2-1.6倍。更广泛地，RL$^V$体现了测试时扩展的协同训练原则：利用强化学习本身产生的数据，共同优化任务表现与推理时有用的能力。
原文摘要 · Abstract (English)
Prevalent reinforcement learning~(RL) methods for fine-tuning LLM reasoners, such as GRPO or Leave-one-out PPO, abandon the learned value function in favor of empirically estimated returns. This hinders test-time compute scaling that relies on using the value-function for verification. Yet if parallel test-time compute is already part of the deployment plan, training should be designed to support it. In this work, we propose RL$^V$ that augments any ``value-free'' RL method by jointly training the LLM as both a reasoner and a generative verifier using RL-generated data, adding verification capabilities without significant overhead. Empirically, RL$^V$ boosts MATH accuracy by over 20\% with parallel sampling and enables $8-32\times$ efficient test-time compute scaling compared to the base RL method. RL$^V$ also exhibits strong generalization capabilities for both easy-to-hard and out-of-domain tasks. Furthermore, RL$^V$ achieves $1.2-1.6\times$ higher performance when jointly scaling parallel and sequential test-time compute with a long reasoning R1 model. More broadly, RL$^V$ instantiates the principle of co-training for test-time scaling: jointly optimizing for task performance and a capability useful at inference, using data that RL training already produces.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。