给大模型推理步骤精准打分,提升逻辑训练效率与泛化能力
Discovering Process-Outcome Credit in Multi-Step LLM Reasoning
- 用逐步信息增益机制量化每步推理价值,过滤训练噪声
- 区分过程与结果奖励,实现推理链与最终答案的解耦优化
- 显著提升样本效率和跨任务泛化能力,适合复杂推理场景
强化学习是提升大语言模型推理能力的有效范式,但传统基于结果的奖励常因奖励稀疏和信用分配低效而受限。本文提出一种新框架,通过引入逐步边际信息增益(MIG)机制,结合单调历史水印,持续量化推理步骤的内在价值,有效过滤训练噪声。为实现解耦信用分配,采用解耦掩码策略,分别对思维链(CoT)施加过程导向奖励,对完整输出施加结果导向奖励。同时引入双门控监督微调(Dual-Gated SFT)目标,以高质结构与事实信号稳定训练。在文本与多模态基准(如MATH、Super-CLEVR)上的大量实验表明,该方法在样本效率与最终准确率上均优于GRPO等基线模型。此外,模型展现出更强的分布外鲁棒性,具备优异的零样本迁移能力,可有效处理未见且复杂的推理任务。
原文摘要 · Abstract (English)
Reinforcement Learning (RL) serves as a potent paradigm for enhancing reasoning capabilities in Large Language Models (LLMs), yet standard outcome-based approaches often suffer from reward sparsity and inefficient credit assignment. In this paper, we propose a novel framework designed to provide continuous reward signals, which introduces a Step-wise Marginal Information Gain (MIG) mechanism that quantifies the intrinsic value of reasoning steps against a Monotonic Historical Watermark, effectively filtering out training noise. To ensure disentangled credit distribution, we implement a Decoupled Masking Strategy, applying process-oriented rewards specifically to the chain-of-thought (CoT) and outcome-oriented rewards to the full completion. Additionally, we incorporate a Dual-Gated SFT objective to stabilize training with high-quality structural and factual signals. Extensive experiments across textual and multi-modal benchmarks (e.g., MATH, Super-CLEVR) demonstrate that our approach consistently outperforms baselines such as GRPO in both sample efficiency and final accuracy. Furthermore, our model exhibits superior out-of-distribution robustness, demonstrating promising zero-shot transfer capabilities to unseen and challenging reasoning tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。