重用历史轨迹让策略梯度算法更快收敛
Reusing Trajectories in Policy Gradients Enables Fast Convergence
- 设计新算法RT-PG，融合最近ω轮的旧轨迹数据
- 理论证明样本复杂度降至˜O(ε⁻²ω⁻¹)，最优可达˜O(ε⁻¹)
- 适合追求高效训练的连续控制任务研究者
策略梯度（PG）方法在连续控制问题中表现优异，但依赖新采样的在线数据，导致样本效率低，需O(ε⁻²)条轨迹才能达到ε近似驻点。现有改进多聚焦于重用梯度，而重用历史轨迹虽直观却缺乏理论支撑。本文首次提供严谨理论证明：重用离线轨迹可显著加速收敛。提出RT-PG算法，采用幂均修正的多重重要性加权估计器，融合最近ω轮的在线与离线数据。通过新分析，证明其样本复杂度为˜O(ε⁻²ω⁻¹)；当重用全部历史轨迹时，可达到˜O(ε⁻¹)，为当前PG方法最优。实验验证了其在基准测试中的有效性。
原文摘要 · Abstract (English)
Policy gradient (PG) methods are a class of effective reinforcement learning algorithms, particularly when dealing with continuous control problems. They rely on fresh on-policy data, making them sample-inefficient and requiring $O(ε^{-2})$ trajectories to reach an $ε$-approximate stationary point. A common strategy to improve efficiency is to reuse information from past iterations, such as previous gradients or trajectories, leading to off-policy PG methods. While gradient reuse has received substantial attention, leading to improved rates up to $O(ε^{-3/2})$, the reuse of past trajectories, although intuitive, remains largely unexplored from a theoretical perspective. In this work, we provide the first rigorous theoretical evidence that reusing past off-policy trajectories can significantly accelerate PG convergence. We propose RT-PG (Reusing Trajectories - Policy Gradient), a novel algorithm that leverages a power mean-corrected multiple importance weighting estimator to effectively combine on-policy and off-policy data coming from the most recent $ω$ iterations. Through a novel analysis, we prove that RT-PG achieves a sample complexity of $\tilde{O}(ε^{-2}ω^{-1})$. When reusing all available past trajectories, this leads to a rate of $\tilde{O}(ε^{-1})$, the best known one in the literature for PG methods. We further validate our approach empirically, demonstrating its effectiveness against baselines with state-of-the-art rates.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。