通过几何约束提升大模型强化训练稳定性,防止性能下降与输出过长。
GCPO: Diagnosing and Constraining Subspace Geometry in Rollout RL for LLMs

- 用主子空间重叠度诊断更新过程中的异常波动
- 在数学推理等任务上比GRPO最高提升27.69分
- 适合追求稳定微调的大模型研究者与工程应用
基于策略的回溯方法(如GRPO)是大语言模型后训练的核心,但常出现训练不稳定、跨任务能力下降和响应长度膨胀等问题。现有工作虽分析了整体更新的子空间结构,但对每一步更新变化及其与性能的关系仍不清晰。本文提出主子空间重叠度,衡量单次回溯更新相对于预训练权重主导奇异子空间的偏离程度。尽管平均重叠度低,但瞬时峰值常预示性能下降。为此,我们提出GCPO(几何约束策略优化),通过硬双边正交投影将更新限制在互补子空间内,从构造上避免异常偏移。在Qwen3-8B和GLM4-9B上,针对数学推理、代码生成和工具使用任务,GCPO持续优于GRPO及近年变体DAPO、GSPO,相比基线模型最高提升27.69分,比最强基线提升2.37分。此外,GCPO保持通用能力,消除响应长度膨胀,稳定策略熵。研究成果为强化学习后训练提供了新的诊断视角与设计原则。
原文摘要 · Abstract (English)
On-policy rollout methods such as GRPO are central to post-training of large language models, yet they frequently suffer from training instabilities, cross-task capability degradation, and response-length inflation. Although prior work has characterized the subspace geometry of aggregate updates, the stepwise variation of this geometry and its relationship to model performance remain unclear. We introduce Principal-Subspace Overlap, a dimension-corrected measure of individual rollout updates relative to the dominant singular subspaces of pretrained weights. Despite low average overlap, transient spikes often precede performance degradation. To address this, we propose GCPO (Geometrically Constrained Policy Optimization), which applies hard bilateral orthogonal projections to constrain updates to the complementary subspaces, preventing such excursions by construction. Across mathematical reasoning, code generation, and tool-use tasks on Qwen3-8B and GLM4-9B, GCPO consistently outperforms GRPO and recent variants, including DAPO and GSPO, improving over the base models and the strongest baseline by up to 27.69 and 2.37 points, respectively. Furthermore, GCPO preserves general capabilities, eliminates response-length inflation, and stabilizes policy entropy. Our findings provide a new diagnostic lens and a principled design perspective for stable reinforcement learning post-training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。