让大模型自己检查并选择性修正答案，提升推理准确性。
PAG: Multi-Turn Reinforced LLM Self-Correction with Policy as Generative Verifier
- 大模型在多轮强化学习中轮流扮演策略和验证者角色。
- 仅当自检发现错误时才修改答案，准确率显著提升。
- 无需额外验证模型，适合需要高可靠推理的场景。
大型语言模型在复杂推理任务中表现优异，但仍难以可靠验证自身输出的正确性。现有解决方案通常依赖独立的验证模型或复杂的多阶段自修正训练流程，限制了可扩展性。本文提出策略即生成验证器（PAG），一种简单有效的框架，使大模型在统一的多轮强化学习范式下通过交替扮演策略与验证者角色实现自我修正。不同于以往始终生成第二次尝试的方法，PAG引入选择性修订机制：仅当模型自身的生成式验证步骤检测到错误时才进行修正。这种先验证后修正的流程不仅缓解了模型坍塌问题，还共同提升了推理与验证能力。在多个推理基准上的大量实验表明，作为策略，PAG在直接生成与自我修正准确率上均有提升；作为验证者，其自验证性能优于自一致性方法。
原文摘要 · Abstract (English)
Large Language Models (LLMs) have demonstrated impressive capabilities in complex reasoning tasks, yet they still struggle to reliably verify the correctness of their own outputs. Existing solutions to this verification challenge often depend on separate verifier models or require multi-stage self-correction training pipelines, which limit scalability. In this paper, we propose Policy as Generative Verifier (PAG), a simple and effective framework that empowers LLMs to self-correct by alternating between policy and verifier roles within a unified multi-turn reinforcement learning (RL) paradigm. Distinct from prior approaches that always generate a second attempt regardless of model confidence, PAG introduces a selective revision mechanism: the model revises its answer only when its own generative verification step detects an error. This verify-then-revise workflow not only alleviates model collapse but also jointly enhances both reasoning and verification abilities. Extensive experiments across diverse reasoning benchmarks highlight PAG's dual advancements: as a policy, it enhances direct generation and self-correction accuracy; as a verifier, its self-verification outperforms self-consistency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。