让大模型自己检查答案对错，提升推理准确率。
Incentivizing LLMs to Self-Verify Their Answers
- 用强化学习让模型同时生成答案并自我验证。
- 在数学推理任务上，后训练性能和推理时扩展效果均提升。
- 无需外部校验器，适合需要高可靠性的场景。
大语言模型在复杂推理任务上取得显著进展，依赖后训练与测试时扩展。然而，仅在特定推理任务上后训练的模型，使用外部奖励模型进行测试时扩展，收益有限。我们发现这源于特定后训练生成器与通用奖励模型之间的分布差异。为此，提出一种激励模型自我验证答案的框架：将答案生成与验证统一于单一强化学习过程，训练出能自主评估自身解题正确性的模型。该模型可在推理时通过自我验证进一步提升表现，无需依赖外部验证器。基于 Qwen2.5-Math-7B 与 DeepSeek-R1-Distill-Qwen-1.5B 训练，实验显示其在多种数学推理基准上不仅提升了后训练性能，还能实现有效测试时扩展。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have demonstrated remarkable progress in complex reasoning tasks through both post-training and test-time scaling laws. While prevalent test-time scaling approaches are often realized by using external reward models to guide the model generation process, we find that only marginal gains can be acquired when scaling a model post-trained on specific reasoning tasks. We identify that the limited improvement stems from distribution discrepancies between the specific post-trained generator and the general reward model. To address this, we propose a framework that incentivizes LLMs to self-verify their own answers. By unifying answer generation and verification within a single reinforcement learning (RL) process, we train models that can effectively assess the correctness of their own solutions. The trained model can further scale its performance at inference time by verifying its generations, without the need for external verifiers. We train our self-verification models based on Qwen2.5-Math-7B and DeepSeek-R1-Distill-Qwen-1.5B, demonstrating their capabilities across varying reasoning context lengths. Experiments on multiple mathematical reasoning benchmarks show that our models can not only improve post-training performance but also enable effective test-time scaling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。