用多阶统计量优化大模型推理,提升成功率与稳定性。
Beyond the Mean: Multi-Moment Policy Optimization for LLM Reasoning

- 将失败概率视为随机变量,同时优化其多个统计矩。
- 在五个数学推理基准上优于现有强基线,表现更稳定。
- 适合关注大模型推理优化的算法研究者和工程师。
强化学习已成为提升大语言模型推理能力的核心范式。现有方法通常聚焦于降低问题间的失败概率。本文从矩的角度重新审视大模型推理的策略优化,将随机采样的问题失败概率视为随机变量,通过其矩来刻画优化目标。多数现有方法仅优化失败概率分布的单一矩,忽略了其整体分布结构。为此,我们提出多矩策略优化(MMPO),联合最小化失败概率分布的多个矩。MMPO可直观理解为最小化首次成功响应所需的期望截断时间。此外,我们进一步构建了通用的矩变换框架,系统性地生成不同矩特征,并统一了更广泛的策略优化目标视角。在五个数学推理基准及不同规模模型上的实验表明,MMPO始终优于强基线。我们希望这一矩视角能为大模型推理的策略优化目标设计提供新思路。
原文摘要 · Abstract (English)
Reinforcement learning has become a central paradigm for improving the reasoning capabilities of large language models. Existing methods generally aim to reduce the failure probabilities induced across problems. In this paper, we introduce a moment-based perspective on policy optimization for LLM reasoning by treating the failure probability of a randomly sampled problem as a random variable and characterizing optimization objectives through its moments. Under this perspective, many existing methods optimize only a single moment of the failure-probability distribution, leaving its broader distributional structure largely uncharacterized. We propose \textbf{M}ulti-\textbf{M}oment \textbf{P}olicy \textbf{O}ptimization (MMPO), a novel policy optimization framework that jointly minimizes multiple moments of the failure-probability distribution. MMPO admits a direct operational interpretation as minimizing the expected truncated time required to obtain the first successful response. Beyond MMPO, we further develop a general moment-transformation framework that systematically induces different moment profiles and provides a unified view of a broader family of policy optimization objectives. Experiments across five mathematical reasoning benchmarks and models of different scales demonstrate that MMPO consistently outperforms strong baselines. We hope this moment-based perspective offers new insights into the design of policy optimization objectives for LLM reasoning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。