arXiv:2512.13070cs.AIcs.CL2025-12中稿 · NeurIPS被引 3

解决大模型自监督强化学习训练崩溃问题,提升推理能力稳定性。

M-GRPO: Stabilizing Self-Supervised Reinforcement Learning for Large Language Models with Momentum-Anchored Policy Optimization

  • 用动量锚定模型提供稳定训练目标,防止策略退化。
  • 通过四分位距动态过滤低熵轨迹,保持策略多样性。
  • 在多个推理任务上实现更稳定训练和顶尖性能,适合长程推理优化。

自监督强化学习为提升大语言模型推理能力提供了无需昂贵人工标注数据的前景。然而我们发现,现有方法在长程训练中存在严重失败模式:策略崩溃导致性能骤降。诊断表明,单纯增加采样数量虽能延缓崩溃,却无法根治。为此,我们提出M-GRPO(动量锚定组相对策略优化)框架,利用缓慢演化的动量模型作为稳定训练目标。同时,我们发现该过程常伴随策略熵急剧下降,导致过早收敛至次优策略。为此,进一步提出基于四分位距(IQR)的自适应过滤方法,动态剔除低熵轨迹以维持策略多样性。在多个推理基准上的广泛实验表明,M-GRPO有效稳定训练过程,IQR过滤防止过早收敛。两项创新结合带来更优的训练稳定性与当前最优性能。

原文摘要 · Abstract (English)

Self-supervised reinforcement learning (RL) presents a promising approach for enhancing the reasoning capabilities of Large Language Models (LLMs) without reliance on expensive human-annotated data. However, we find that existing methods suffer from a critical failure mode under long-horizon training: a "policy collapse" where performance precipitously degrades. We diagnose this instability and demonstrate that simply scaling the number of rollouts -- a common strategy to improve performance -- only delays, but does not prevent, this collapse. To counteract this instability, we first introduce M-GRPO (Momentum-Anchored Group Relative Policy Optimization), a framework that leverages a slowly evolving momentum model to provide a stable training target. In addition, we identify that this process is often accompanied by a rapid collapse in policy entropy, resulting in a prematurely confident and suboptimal policy. To specifically address this issue, we propose a second contribution: an adaptive filtering method based on the interquartile range (IQR) that dynamically prunes low-entropy trajectories, preserving essential policy diversity. Our extensive experiments on multiple reasoning benchmarks demonstrate that M-GRPO stabilizes the training process while the IQR filter prevents premature convergence. The combination of these two innovations leads to superior training stability and state-of-the-art performance.

强化学习大模型训练稳定推理优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。