提出无剪裁的策略优化方法，解决大模型强化训练中的不稳定问题。
Clipping-Free Policy Optimization for Large Language Models
- 用凸二次惩罚替代启发式剪裁，实现全程可微的目标函数。
- 在推理和对齐任务中均保持性能，且训练更稳定、减少过度生成。
- 无需额外超参数，仅需一行代码即可替换现有方法，适合快速部署。
强化学习已成为大语言模型后训练的核心，但主流算法依赖剪裁机制，在大规模场景下引入优化问题，包括零梯度区域、奖励滥用和训练不稳定性。我们提出无剪裁策略优化（CFPO），以总变差发散约束导出的凸二次惩罚取代启发式剪裁，获得全程可微的目标函数，实现无硬边界下的稳定策略更新。我们在推理与对齐设置中评估了CFPO。在推理任务中，CFPO在下游基准上表现与剪裁方法相当，并扩展了稳定训练区间；在对齐任务中，有效缓解冗长输出滥用问题，降低能力退化，同时保持竞争力的指令遵循性能。CFPO仅需一行代码修改且无需额外超参数。结果表明，其是剪裁类方法在大模型后训练中的可靠替代方案。
原文摘要 · Abstract (English)
Reinforcement learning has become central to post-training large language models, yet dominant algorithms rely on clipping mechanisms that introduce optimization issues at scale, including zero-gradient regions, reward hacking, and training instability. We propose Clipping-Free Policy Optimization (CFPO), which replaces heuristic clipping with a convex quadratic penalty derived from Total Variation divergence constraints, yielding an everywhere-differentiable objective that enforces stable policy updates without hard boundaries. We evaluate CFPO across both reasoning and alignment settings. In reasoning, CFPO matches clipping-based methods on downstream benchmarks while extending the stable training regime. In alignment, CFPO mitigates verbosity exploitation and reduces capability degradation, while achieving competitive instruction-following performance. CFPO requires only a one-line code change and no additional hyperparameters. Our results suggest that CFPO is a promising drop-in alternative to clipping-based methods for LLM post-training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。