CSAC算法提升强化学习控制的稳定性和样本效率。
Effective Reinforcement Learning Control using Conservative Soft Actor-Critic
- 结合熵与相对熵正则化，平衡探索与更新稳定性。
- 在基准任务和机器人仿真中显著提升学习效率与鲁棒性。
- 适合动态环境下需要高效稳定控制的智能体应用。
强化学习在复杂控制任务中展现出巨大潜力，尤其当与深度神经网络结合使用时。然而，在实际应用中，如何平衡探索、学习稳定性和样本效率仍是一大挑战。传统方法如软演员-评论家（SAC）和近端策略优化（PPO）通过引入熵或相对熵正则化来缓解这些问题，但常面临不稳定和样本效率低的困境。本文提出保守软演员-评论家（CSAC）算法，将熵与相对熵正则化无缝整合到演员-评论家框架中。该方法通过熵正则化增强探索能力，同时利用相对熵正则化避免过激的策略更新。在基准任务和真实机器人仿真中的评估表明，相较于现有方法，CSAC在稳定性和效率方面均有显著提升。这些结果表明，CSAC在动态环境下的控制任务中具备强鲁棒性和应用潜力。
原文摘要 · Abstract (English)
Reinforcement Learning (RL) has shown great potential in complex control tasks, particularly when combined with deep neural networks within the Actor-Critic (AC) framework. However, in practical applications, balancing exploration, learning stability, and sample efficiency remains a significant challenge. Traditional methods such as Soft Actor-Critic (SAC) and Proximal Policy Optimization (PPO) address these issues by incorporating entropy or relative entropy regularization, but often face problems of instability and low sample efficiency. In this paper, we propose the Conservative Soft Actor-Critic (CSAC) algorithm, which seamlessly integrates entropy and relative entropy regularization within the AC framework. CSAC improves exploration through entropy regularization while avoiding overly aggressive policy updates with the use of relative entropy regularization. Evaluations on benchmark tasks and real-world robotic simulations demonstrate that CSAC offers significant improvements in stability and efficiency over existing methods. These findings suggest that CSAC provides strong robustness and application potential in control tasks under dynamic environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。