arXiv:2605.01327cs.AIcs.LG2026-05被引 2

让强化学习按推理步骤更新,提升多模态推理的准确性和稳定性。

Segment-Aligned Policy Optimization for Multi-Modal Reasoning

论文配图:Segment-Aligned Policy Optimization for Multi-Modal Reasoning
图 1 · 摘自论文原文
  • 以推理段落为单位进行策略优化,更贴合人类思考过程。
  • 在多个基准测试中显著提升准确率,训练更稳定。
  • 适合需要精细推理控制的AI系统开发者。

现有大语言模型的强化学习方法通常在单个词元或完整回复序列层面进行策略优化,但此类设定常与推理过程的自然分步结构不匹配,导致信用分配不佳和多模态推理任务中训练不稳定。为此,我们提出段落对齐策略优化(SAPO),一种新型强化学习范式,将连贯的推理步骤而非词元或完整序列作为策略更新的基本单元。SAPO在推理段落上引入分步马尔可夫决策过程抽象,并配备段落级价值估计、优势计算和重要性采样机制,语义上与推理边界对齐。在代表性推理基准上的实验表明,SAPO持续优于词元级和序列级策略优化方法,在保持更好训练稳定性的同时显著提升准确率,并实现更一致的价值估计。本工作强调了将强化学习更新与推理内在结构对齐的重要性,为复杂推理任务中更高效、语义合理的策略优化提供了新路径。代码与模型将公开以确保完全可复现。

原文摘要 · Abstract (English)

Existing reinforcement learning approaches for Large Language Models typically perform policy optimization at the granularity of individual tokens or entire response sequences. However, such formulations often misalign with the natural step-wise structure of reasoning processes, leading to suboptimal credit assignment and unstable training in multi-modal reasoning tasks. To bridge this gap, we propose Segment-Aligned Policy Optimization (SAPO), a novel reinforcement learning paradigm that treats coherent reasoning steps, rather than tokens or full sequences as fundamental units of policy update. SAPO introduces a step-wise Markov decision process abstraction over reasoning segments, accompanied by segment-level value estimation, advantage computation, and importance sampling mechanisms that are semantically aligned with reasoning boundaries. Experiments on representative reasoning benchmarks demonstrate that SAPO consistently outperforms token-level and sequence-level policy optimization methods, achieving significant accuracy improvements while exhibiting better training stability and value estimation consistency. Our work underscores the importance of aligning reinforcement learning updates with the intrinsic structure of reasoning, paving the way for more efficient and semantically grounded policy optimization in complex reasoning tasks. Codes and models will be released to ensure full reproducibility.

强化学习推理优化大模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。