通过优化行为策略降低强化学习梯度方差，提升训练效率与稳定性。
Actor-Critic with Active Importance Sampling
- 用重要性采样思想动态优化行为策略以减少梯度方差
- 在倒立摆和半猫鼠任务中实现更快收敛和更高样本效率
- 适合追求稳定高效强化学习训练的科研与工程应用
本文提出主动重要性采样演员-评论家（AISAC）算法，扩展了演员-评论家框架以降低策略梯度估计的方差。AISAC 通过优化行为策略来最小化梯度方差，同时保持梯度估计的无偏性。基于重要性采样原理，该算法将行为策略调整至与目标策略梯度对齐的高效数据收集分布。针对连续动作空间，AISAC 采用通过交叉熵最小化优化的高斯行为策略。理论分析证明了其方差降低与无偏性。在倒立摆（Inverted Pendulum）和半猫鼠（Half Cheetah）任务上的实验表明，相比标准演员-评论家方法，AISAC 具有更快的学习速度、更高的样本效率和更稳定的训练过程。结果还显示，优化行为策略可提升目标策略更新质量及评论家估计精度，在不同超参数设置下均表现良好。AISAC 加速了收敛并稳定了强化学习训练，具有实际应用潜力。未来工作将探索与 Soft Actor-Critic 及 TD3 等先进算法结合，应用于更复杂环境。
原文摘要 · Abstract (English)
This paper introduces the Active-Importance-Sampling Actor-Critic (AISAC) algorithm, an extension of the Actor-Critic framework for reducing variance in policy gradient estimation. AISAC optimizes the behavior policy to minimize gradient variance while preserving unbiased gradient estimates. Using importance sampling principles, the algorithm adapts the behavior policy toward efficient data collection distributions aligned with target policy gradients. For continuous action spaces, AISAC employs Gaussian behavior policies optimized through cross-entropy minimization. We provide theoretical analysis demonstrating variance reduction and unbiasedness. Experiments on Inverted Pendulum and Half Cheetah tasks show improved learning speed, sample efficiency, and training stability compared to standard Actor-Critic methods. Results indicate that optimizing the behavior policy improves both target policy updates and critic estimation accuracy across different hyperparameter settings. AISAC accelerates convergence and stabilizes reinforcement learning training, making it promising for real-world applications. Future work includes integration with advanced algorithms such as Soft Actor-Critic and TD3 for more complex environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。