提出改进的单时标策略梯度算法，样本效率提升至最优阶
Optimal Sample Complexity for Single Time-Scale Actor-Critic with Momentum
- 用STORM降低评论家更新方差，结合最近样本缓存增强稳定性
- 实现$O(ε^{-2})$样本复杂度，优于此前$O(ε^{-3})$的最优结果
- 适用于深度强化学习框架，仅需微小修改即可落地
在有限状态-动作空间的无限时域折扣马尔可夫决策过程（MDPs）中，我们建立了使用单时标演员-评论家（AC）算法获得ε-最优全局策略的最优样本复杂度为$O(ε^{-2})$，优于此前$O(ε^{-3})$的最优水平。该方法将STORM（STOchastic Recursive Momentum）应用于评论家更新以降低方差，但由于采样来自随策略演进而变化的非平稳占据分布，仅靠STORM仍不足。为此，我们维护一个近期样本的小缓冲区，并对每个评论家更新进行均匀采样。关键的是，这些机制与现有深度学习架构兼容，只需少量修改，不损害实用性。
原文摘要 · Abstract (English)
We establish an optimal sample complexity of $O(ε^{-2})$ for obtaining an $ε$-optimal global policy using a single-timescale actor-critic (AC) algorithm in infinite-horizon discounted Markov decision processes (MDPs) with finite state-action spaces, improving upon the prior state of the art of $O(ε^{-3})$. Our approach applies STORM (STOchastic Recursive Momentum) to reduce variance in the critic updates. However, because samples are drawn from a nonstationary occupancy measure induced by the evolving policy, variance reduction via STORM alone is insufficient. To address this challenge, we maintain a buffer of small fraction of recent samples and uniformly sample from it for each critic update. Importantly, these mechanisms are compatible with existing deep learning architectures and require only minor modifications, without compromising practical applicability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。