用随机化方法高效计算自然策略梯度，无需复杂矩阵运算。
Randomized Advantage Transformation (RAT): Computing Natural Policy Gradients via Direct Backpropagation
- 通过变换优势函数，直接反向传播计算正则化自然梯度。
- 在连续控制和视觉控制任务中性能媲美或超越传统自然梯度方法。
- 实现简单且兼容多种网络结构，适合实际强化学习应用。
自然策略梯度通过考虑分布空间的几何结构提升优化效果，但其实际应用受限于费时的费舍尔矩阵估计与求逆。本文提出随机优势变换（RAT），通过直接反向传播估算带蒂科诺夫正则化的自然策略梯度。利用Woodbury公式，将正则化自然梯度重述为带有变换优势的普通策略梯度。RAT通过在在线小批量上进行随机块卡茨马尔兹迭代高效计算该变换，避免了显式构建费舍尔矩阵、共轭梯度求解器及架构特定近似。我们提供了RAT的收敛性保证，并在连续控制和视觉控制基准上实证表明，其性能达到或超过现有自然梯度方法，同时实现简单易用且兼容多种网络架构。
原文摘要 · Abstract (English)
Natural policy gradients improve optimization by accounting for the geometry of distribution space, but their practical use is limited by the cost of estimating and inverting the Fisher matrix. We present Randomized Advantage Transformation (RAT), a method for estimating Tikhonov-regularized natural policy gradients via direct backpropagation. By applying the Woodbury formula, we reformulate the regularized natural policy gradients as vanilla policy gradients with a transformed advantage. RAT computes this transformation efficiently via randomized block Kaczmarz iterations on on-policy mini-batches, avoiding explicit Fisher construction, conjugate-gradient solvers, and architecture-specific approximations. We provide convergence guarantees for RAT and demonstrate empirically that it matches or exceeds established natural-gradient methods across continuous and visual control benchmarks, while remaining simple to implement and compatible with various architectures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。