arXiv:2505.13445cs.AIcs.CL2025-05NeurIPS被引 42

让大模型自己验证答案,提升推理可靠性。

Trust, But Verify: A Self-Verification Approach to Reinforcement Learning with Verifiable Rewards

  • 用在线验证反馈同步训练模型解题与自检能力。
  • 在数学推理任务上准确率显著提升,自检更频繁准确。
  • 适合需要高可靠性的智能系统开发人员参考。

大型语言模型在复杂推理中展现出巨大潜力,而基于可验证奖励的强化学习(RLVR)是关键改进策略。然而,普遍存在“表面自我反思”问题,即模型无法稳健验证自身输出。本文提出RISE(通过自验证强化推理),一种新型在线强化学习框架,旨在解决该问题。RISE在单一整合的强化学习过程中,显式且同步地训练大模型提升其问题求解与自验证能力。核心机制是利用结果验证器提供的可验证奖励,为解题生成和自验证任务提供实时反馈。每次迭代中,模型先生成解决方案,再批判性评估自身生成的解题路径,两者共同参与策略更新。在多个数学推理基准上的大量实验表明,RISE不仅持续提升模型的问题求解准确率,还显著增强自验证能力。分析显示,在线验证与更高验证计算量带来明显优势。此外,RISE模型在推理过程中表现出更频繁且更准确的自验证行为。这些优势使RISE成为构建更鲁棒、更具自我意识推理者的一条灵活有效路径。

原文摘要 · Abstract (English)

Large Language Models (LLMs) show great promise in complex reasoning, with Reinforcement Learning with Verifiable Rewards (RLVR) being a key enhancement strategy. However, a prevalent issue is ``superficial self-reflection'', where models fail to robustly verify their own outputs. We introduce RISE (Reinforcing Reasoning with Self-Verification), a novel online RL framework designed to tackle this. RISE explicitly and simultaneously trains an LLM to improve both its problem-solving and self-verification abilities within a single, integrated RL process. The core mechanism involves leveraging verifiable rewards from an outcome verifier to provide on-the-fly feedback for both solution generation and self-verification tasks. In each iteration, the model generates solutions, then critiques its own on-policy generated solutions, with both trajectories contributing to the policy update. Extensive experiments on diverse mathematical reasoning benchmarks show that RISE consistently improves model's problem-solving accuracy while concurrently fostering strong self-verification skills. Our analyses highlight the advantages of online verification and the benefits of increased verification compute. Additionally, RISE models exhibit more frequent and accurate self-verification behaviors during reasoning. These advantages reinforce RISE as a flexible and effective path towards developing more robust and self-aware reasoners.

强化学习自验证大模型推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。