PPO的裁剪损失其实等价于可变系数的KL惩罚，揭示了其内在机制。
KLip-PPO: A per-sample KL perspective on PPO-Clip
- 用随样本变化的KL惩罚重构PPO裁剪损失，数学上完全等价。
- 在五个连续控制任务中，新旧方法训练曲线几乎完全一致。
- 揭示了裁剪机制的本质是信任区域边界处的阶跃惩罚，利于算法改进。
近端策略优化（PPO）是当前主流的在线策略强化学习算法。现有文献将其呈现为两种形式：一种是通过裁剪代理函数限制策略间重要性比值，另一种是施加两者之间的Kullback-Leibler（KL）惩罚。这两种形式被视为独立算法，拥有各自的梯度、超参数和参考实现，并有大量实证研究进行比较。本文证明，裁剪代理函数的梯度可被一个系数随样本变化的KL代理函数精确重现，其系数具有关于重要性比值和优势的闭式表达。该等式在每个小批量步骤和整个内循环中均成立。在五个MuJoCo连续控制基准测试中，两种损失产生的训练曲线无法区分。这一重述揭示了裁剪代理函数隐藏的结构性特征：PPO-Clip的隐式每样本惩罚在信任区域边界处表现为阶跃函数，其系数形状是推广算法的自然设计轴线。讨论部分简要勾勒了由此衍生的后续方向。
原文摘要 · Abstract (English)
Proximal Policy Optimization (PPO) is the standard policy-gradient algorithm for on-policy reinforcement learning. The literature presents it in two forms, a clipped surrogate that bounds the importance ratio between successive policies and a Kullback-Leibler penalty between them. These forms are treated as separate algorithms with their own gradients, their own hyperparameters, and their own reference implementations, and a sizeable body of empirical work compares them. We show that the gradient of the clipped surrogate is reproduced exactly by a Kullback-Leibler surrogate whose coefficient varies per sample, with closed-form dependence on the importance ratio and the advantage. The identity holds at every minibatch step and across the entire inner loop, and on five MuJoCo continuous-control benchmarks the two losses produce indistinguishable training curves. The reformulation exposes a structural feature of the clipped surrogate that the min notation hides. PPO-Clip's implicit per-sample penalty is a step function at the boundary of the trust region, and the shape of this coefficient is the natural design axis for generalising the algorithm. We sketch the resulting follow-up directions in the discussion.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。