提出新方法BandPO，解决大模型强化学习中探索不足问题
BandPO: Bridging Trust Regions and Ratio Clipping via Probability-Aware Bounds for LLM Reinforcement Learning
- 用动态概率感知的区间替代固定剪裁，更好平衡探索与稳定
- 在多个模型和数据集上显著优于传统剪裁方法，熵衰减更慢
- 理论可证明最优解，适合需要稳定训练的大模型强化学习
近端约束是大语言模型强化学习稳定性的基础。虽然PPO中的经典剪裁机制作为信任区域的有效代理，但存在关键瓶颈：固定边界严格限制低概率动作的向上更新幅度，过度抑制高优势尾部策略，导致熵快速下降。为此，我们提出带约束策略优化（BandPO）。BandPO用统一的理论算子Band替代经典剪裁，将由f-散度定义的信任区域投影为动态的概率感知剪裁区间。理论分析表明，Band能有效解决此探索瓶颈。我们将该映射建模为凸优化问题，保证全局最优数值解，并推导出特定散度的闭式解。在多种模型和数据集上的大量实验表明，BandPO始终优于经典剪裁和Clip-Higher，且能稳健缓解熵崩溃。
原文摘要 · Abstract (English)
Proximal constraints are fundamental to the stability of the Large Language Model reinforcement learning. While the canonical clipping mechanism in PPO serves as an efficient surrogate for trust regions, we identify a critical bottleneck: fixed bounds strictly constrain the upward update margin of low-probability actions, disproportionately suppressing high-advantage tail strategies and inducing rapid entropy collapse. To address this, we introduce Band-constrained Policy Optimization (BandPO). BandPO replaces canonical clipping with Band, a unified theoretical operator that projects trust regions defined by f-divergences into dynamic, probability-aware clipping intervals. Theoretical analysis confirms that Band effectively resolves this exploration bottleneck. We formulate this mapping as a convex optimization problem, guaranteeing a globally optimal numerical solution while deriving closed-form solutions for specific divergences. Extensive experiments across diverse models and datasets demonstrate that BandPO consistently outperforms canonical clipping and Clip-Higher, while robustly mitigating entropy collapse.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。