通过筛选有效经验降低策略梯度方差，提升强化学习样本效率
Variance Reduction Based Experience Replay for Policy Optimization
- 按信息量选择重用经验，减少梯度估计方差
- 在多个环境上加速学习，超越现有最优算法
- 理论证明存在偏差-方差权衡，适合高样本成本场景
复杂随机系统中的高效强化学习需要利用历史数据以提升样本效率并加速策略优化。然而，经典经验回放对所有过往观测一视同仁，未能考虑其对学习贡献的差异。为此，我们提出方差缩减经验回放（VRER），一种可选择性重用信息丰富样本的原理性框架，以降低策略梯度估计的方差。该方法与算法无关，可集成至现有策略优化方法中，形成样本高效的离策略算法——带VRER的策略梯度（PG-VRER）。为提供严格的理论保证，我们构建了新的经验回放分析框架，明确考虑马尔可夫动态和行为策略交互带来的依赖关系。基于此，我们建立了PG-VRER的有限时间收敛性保障，并揭示了一个基本的偏差-方差权衡：重用旧样本可降低梯度方差，但可能引入更大估计偏差。大量实验表明，VRER持续加速学习过程，并优于当前最优策略优化算法。
原文摘要 · Abstract (English)
Effective reinforcement learning (RL) for complex stochastic systems requires leveraging historical data to improve sample efficiency and accelerate policy optimization. However, classical experience replay treats all past observations uniformly and fails to account for their varying contributions to learning. To address this limitation, we propose Variance Reduction Experience Replay (VRER), a principled framework that selectively reuses informative samples to reduce the variance of policy gradient estimates. VRER is algorithm-agnostic and can be integrated with existing policy optimization methods, yielding the sample-efficient off-policy algorithm, Policy Gradient with VRER (PG-VRER). To provide rigorous theoretical guarantees, we develop a novel analysis framework for experience replay that explicitly accounts for dependencies induced by Markovian dynamics and behavior-policy interactions. Using this framework, we establish finite-time convergence guarantees for PG-VRER and characterize a fundamental bias-variance trade-off: reusing older samples reduces gradient variance but may introduce greater estimation bias. Extensive experiments show that VRER consistently accelerates learning and outperforms state-of-the-art policy optimization algorithms
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。