让大模型既答对又理得清,通过验证中间推理步骤提升可靠性。
Correct Answers from Sound Reasoning: Verifiable Process Supervision for Language Models
- 监督中间推理步骤,用可验证的环节奖励优化过程质量。
- 在国际象棋中,推理错误减少30%,一致性接近饱和。
- 适合需要可靠逻辑链的任务,如数学、复杂决策等场景。
训练语言模型同时产出正确答案和合理推理仍是挑战。传统基于可验证奖励的强化学习仅优化最终结果,可能提高准确率却损害推理质量,导致答案不准确、不完整或不一致。本文提出可验证过程监督(VPS),一种后训练框架,通过监督结构化中间结论,联合优化预测准确率与推理质量。首先通过监督微调引导模型生成结构化推理,实现中间结论的确定性提取与验证,用于过程级奖励。针对推理子任务难度差异,引入自适应加权机制,优先关注误差最大的部分,形成隐式课程。在国际象棋这一可控测试平台中,模型推理可被引擎信号精确验证。结果显示,仅优化结果的强化学习虽提升走法准确率,但推理质量急剧下降,胜率误差最高上升112%,内部一致性最高下降69%。而VPS在保持准确率的同时显著提升推理质量,胜率误差降低最多达30%,一致性恢复至接近饱和。推理空间分析表明,缺乏结构先验时,仅优化结果的强化学习会收敛到依赖预算的捷径,而非多步严谨推理。在数学推理任务中亦观察到类似现象:仅优化结果的方法提升准确率但恶化步骤级算术与一致性;而VPS则同时维持两者。结果证明,VPS使语言模型在可验证领域实现既准确又可靠的推理。
原文摘要 · Abstract (English)
Training language models to produce both correct answers and sound reasoning remains an open challenge. Reinforcement learning with verifiable rewards typically optimizes only final outcomes, which can improve task accuracy at the expense of reasoning quality, producing inaccurate, incomplete, or inconsistent traces. We propose verifiable process supervision (VPS), a post-training framework that jointly optimizes prediction accuracy and reasoning quality by supervising structured intermediate claims. We first apply supervised fine-tuning to induce a structured reasoning format, enabling deterministic extraction and verification of intermediate claims for process-level rewards. To address the heterogeneous difficulty of reasoning subtasks, we introduce adaptive weighting that prioritizes components with the largest remaining errors, creating an implicit curriculum. We evaluate VPS on chess as a controlled testbed where reasoning steps can be deterministically verified against engine signals. While outcome-only RL improves move accuracy, it sharply degrades reasoning quality, increasing win-rate error by up to 112% and reducing internal consistency by up to 69%. In contrast, VPS preserves accuracy while significantly improving reasoning quality, reducing win-rate error by up to 30% and restoring consistency to near saturation. A reasoning-space analysis further shows that, without a structured prior, outcome-only RL converges to budget-dependent shortcuts rather than sound multi-step reasoning. Beyond chess, we observe the same phenomenon on math reasoning, where outcome-only RL improves accuracy while degrading step-level arithmetic and consistency, whereas VPS maintains both. These results show that VPS enables language models to reason both accurately and reliably in verifiable domains.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。