用概率平滑替代传统截断，让大模型强化学习更稳定且能持续优化。
It's Not You, It's Clipping: A Soft Trust-Region via Probability Smoothing for LLM RL
- 通过线性插值平滑策略概率，构建软信任区域
- 在GSM8K上达79.9%准确率，优于传统方法
- 适合需要精细微调的数学推理类模型
使用PPO和GRPO等强化学习方法训练大语言模型时，通常依赖比率截断来稳定更新。尽管有效防止了不稳定性，但截断会丢弃信息、引入梯度不连续性，并阻碍对更优策略的探索。受标签平滑启发，我们提出概率平滑策略优化（PSPO）。PSPO在计算重要性比率前，将当前策略概率向行为策略平滑，形成一个保留梯度的软信任区域。与依赖sigmoid变换的先前软截断方法不同，我们的方法采用线性插值，提供更简单、更鲁棒的梯度保持能力。实验表明，GR-PSPO在基于先验领域知识微调模型时，优于截断和sigmoid基方法，在GSM8K上达到79.9%准确率，在MATH上达到59.6%准确率，适用于Qwen2-Math-1.5B模型。
原文摘要 · Abstract (English)
Training large language models (LLMs) with reinforcement learning (RL) methods such as PPO and GRPO commonly relies on ratio clipping to stabilise updates. While effective at preventing instability, clipping discards information, introduces gradient discontinuities and can prevent exploration of better policies. Inspired by label smoothing, we propose Probability Smoothing Policy Optimisation (PSPO). PSPO smooths current policy probabilities toward the behaviour policy before computing importance ratios, creating a soft trust region that preserves gradients while preventing destabilising updates. Unlike prior soft clipping approaches that use sigmoid-based transformations which can suffer from vanishing gradients and saturation, our method uses a linear interpolation, providing simpler and more robust gradient preservation. Empirically, GR-PSPO outperforms clipping and sigmoid-based alternatives on mathematical reasoning benchmarks when refining models with prior domain knowledge, achieving an accuracy of 79.9% on GSM8K and 59.6% on MATH for Qwen2-Math-1.5B.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。