ISOPO用单步梯度实现自然策略梯度，无需旧策略比较。
ISOPO: Proximal policy gradients without pi-old
- 通过费舍尔度量归一化序列梯度，一步逼近自然策略梯度。
- 相比传统方法，仅需一次反向传播即可完成更新，计算开销极低。
- 适合追求高效强化学习训练的从业者，尤其适用于在线策略优化场景。
本文提出等距策略优化（ISOPO），一种在单次梯度步骤中高效近似自然策略梯度的方法。与现有近端策略方法（如GRPO或CISPO）不同，后者需多次梯度步骤并采用重要性比率裁剪来近似相对于参考策略的自然梯度步，ISOPO在最简形式下，先在费舍尔度量中对每个序列的对数概率梯度进行归一化，再与优势函数收缩。另一种变体则基于各层的神经正切核对微批次优势进行变换，该变换以逐层方式在单次反向传播中完成，相较于标准REINFORCE，计算开销几乎可忽略。
原文摘要 · Abstract (English)
This note introduces Isometric Policy Optimization (ISOPO), an efficient method to approximate the natural policy gradient in a single gradient step. In comparison, existing proximal policy methods such as GRPO or CISPO use multiple gradient steps with variants of importance ratio clipping to approximate a natural gradient step relative to a reference policy. In its simplest form, ISOPO normalizes the log-probability gradient of each sequence in the Fisher metric before contracting with the advantages. Another variant of ISOPO transforms the microbatch advantages based on the neural tangent kernel in each layer. ISOPO applies this transformation layer-wise in a single backward pass and can be implemented with negligible computational overhead compared to vanilla REINFORCE.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。