用对数障碍项增强策略优化中的探索能力，提升算法鲁棒性。
How Log-Barrier Helps Exploration in Policy Optimization
- 在策略优化目标中加入对数障碍正则，强制保持探索
- 无需假设最优动作概率远离零，仍能收敛
- 理论与仿真均验证其在无理想假设下的有效性
近期研究表明，使用固定学习率的随机梯度带通（SGB）算法可收敛至全局最优策略。但该结论依赖于不切实际的假设：最优动作的概率始终远离零。我们将其归因于SGB缺乏显式的探索机制。为此，我们提出在SGB目标中引入对参数化策略的对数障碍正则，结构化地强制最小探索量。我们证明，对数障碍随机梯度带通（LB-SGB）在样本复杂度上与SGB相当，且在无任何学习过程假设下仍能收敛（速率较慢）。我们还揭示了对数障碍正则与自然策略梯度之间的联系：二者均通过控制Fisher信息来利用策略空间的几何结构。数值模拟验证了理论结果，展示了对数障碍正则的优势。
原文摘要 · Abstract (English)
Recently, it has been shown that the Stochastic Gradient Bandit (SGB) algorithm converges to a globally optimal policy with a constant learning rate. However, these guarantees rely on unrealistic assumptions about the learning process, namely that the probability of the optimal action is always bounded away from zero. We attribute this to the lack of an explicit exploration mechanism in SGB. To address these limitations, we propose to regularize the SGB objective with a log-barrier on the parametric policy, structurally enforcing a minimal amount of exploration. We prove that Log-Barrier Stochastic Gradient Bandit (LB-SGB) matches the sample complexity of SGB, but also converges (at a slower rate) without any assumptions on the learning process. We also show a connection between the log-barrier regularization and Natural Policy Gradient, as both exploit the geometry of the policy space by controlling the Fisher information. We validate our theoretical findings through numerical simulations, showing the benefits of the log-barrier regularization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。