改进强化学习算法SAC，使其稳定使用多步回报提升训练速度。
SACn: Soft Actor-Critic with n-step Returns
- 设计新方法实现SAC与多步回报的稳定结合，避免数值不稳。
- 提出τ采样熵估计，降低学习目标方差，提升收敛性。
- 在MuJoCo仿真环境中验证，显著加速训练且性能更优。
Soft Actor-Critic（SAC）是当前主流的离线策略、在线无模型强化学习方法，广泛应用于实际场景。相比单步回报，多步回报可加快算法收敛速度。然而，将多步回报与SAC结合会因动作分布变化引入偏差，传统解决方案依赖重要性采样，但可能导致数值不稳定。本文提出一种数值稳定的改进型重要性采样方法，简化超参数选择。同时，在多步最大熵框架下分析并优化了熵估计，提出τ-采样熵估计以减少学习目标方差。最终构建SACn算法，并在MuJoCo仿真环境中进行实验验证，结果表明该方法能有效提升训练效率与稳定性。
原文摘要 · Abstract (English)
Soft Actor-Critic (SAC) is widely used in practical applications and is now one of the most relevant off-policy online model-free reinforcement learning (RL) methods. The technique of n-step returns is known to increase the convergence speed of RL algorithms compared to their 1-step returns-based versions. However, SAC is notoriously difficult to combine with n-step returns, since their usual combination introduces bias in off-policy algorithms due to the changes in action distribution. While this problem is solved by importance sampling, a method for estimating expected values of one distribution using samples from another distribution, importance sampling may result in numerical instability. In this work, we combine SAC with n-step returns in a way that overcomes this issue. We present an approach to applying numerically stable importance sampling with simplified hyperparameter selection. Furthermore, we analyze the entropy estimation approach of Soft Actor-Critic in the context of the n-step maximum entropy framework and formulate the $τ$-sampled entropy estimation to reduce the variance of the learning target. Finally, we formulate the Soft Actor-Critic with n-step returns (SAC$n$) algorithm that we experimentally verify on MuJoCo simulated environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。