双向软演员-评论家算法提升强化学习效率
Bidirectional Soft Actor-Critic: Leveraging Forward and Reverse KL Divergence for Efficient Reinforcement Learning
- 结合前向与反向KL散度，先显式初始化策略
- 在连续控制任务中奖励提升最高达30%
- 适合追求高样本效率的强化学习研究者
软演员-评论家（SAC）是最大熵强化学习中的先进方法，传统上依赖最小化反向Kullback-Leibler（KL）散度进行策略更新。然而，这种方法导致最优投影策略不可解析，需梯度近似，易引发不稳定性与低样本效率。本文研究在SAC中使用前向KL散度的可行性。对于高斯策略，我们证明前向KL可导出显式最优投影策略——对应目标Boltzmann分布动作边际的均值与方差。基于两种KL方向的优势，提出双向SAC：先用前向KL显式初始化策略，再通过优化反向KL进行精调。在连续控制基准测试中，双向SAC显著优于标准SAC及其他基线，平均奖励提升达30%，且样本效率更高。
原文摘要 · Abstract (English)
The Soft Actor-Critic (SAC) algorithm, a state-of-the-art method in maximum entropy reinforcement learning, traditionally relies on minimizing reverse Kullback-Leibler (KL) divergence for policy updates. However, this approach leads to an intractable optimal projection policy, necessitating gradient-based approximations that can suffer from instability and poor sample efficiency. This paper investigates the alternative use of forward KL divergence within SAC. We demonstrate that for Gaussian policies, forward KL divergence yields an explicit optimal projection policy -- corresponding to the mean and variance of the target Boltzmann distribution's action marginals. Building on the distinct advantages of both KL directions, we propose Bidirectional SAC, an algorithm that first initializes the policy using the explicit forward KL projection and then refines it by optimizing the reverse KL divergence. Comprehensive experiments on continuous control benchmarks show that Bidirectional SAC significantly outperforms standard SAC and other baselines, achieving up to a $30\%$ increase in episodic rewards, alongside enhanced sample efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。