纠正SAC在高维动作空间中的分布偏移,提升强化学习性能
Rethinking Soft Actor-Critic in High-Dimensional Action Spaces: The Cost of Ignoring Distribution Shift
- 引入tanh变换的精确概率密度函数,揭示动作分布失真机制
- 在HumanoidBench上实现奖励显著提升与采样效率优化
- 适用于高维连续控制任务,对SAC类算法有重要改进价值
Soft Actor-Critic(SAC)在多种深度强化学习任务中表现稳健,其通过tanh变换将动作限制在有界范围内。然而,该变换引发分布偏移,扭曲原始高斯动作分布,导致策略在高维动作空间中选择次优动作。本文开展理论与实证分析,推导tanh变换后动作的精确概率密度函数(PDF),阐明变换后分布的众数与期望输出之间的偏差。在HumanoidBench基准的高维任务上进行大量实验,结果表明:考虑该分布偏移可显著提升SAC的累积奖励、样本效率和可靠性。研究强调:在高维深度强化学习环境中,应对变换引起的分布偏移予以重视,以增强SAC等算法的鲁棒性与适用性。
原文摘要 · Abstract (English)
Soft Actor-Critic algorithm is widely recognized for its robust performance across a range of deep reinforcement learning tasks, where it leverages the tanh transformation to constrain actions within bounded limits. However, this transformation induces a distribution shift, distorting the original Gaussian action distribution and potentially leading the policy to select suboptimal actions, particularly in high-dimensional action spaces. In this paper, we conduct a comprehensive theoretical and empirical analysis of this distribution shift, deriving the precise probability density function (PDF) for actions following the tanh transformation to clarify the misalignment introduced between the transformed distribution's mode and the intended action output. We substantiate these theoretical insights through extensive experiments on high-dimensional tasks within the HumanoidBench benchmark. Our findings indicate that accounting for this distribution shift substantially enhances SAC's performance, resulting in notable improvements in cumulative rewards, sample efficiency, and reliability across tasks. These results underscore a critical consideration for SAC and similar algorithms: addressing transformation-induced distribution shifts is essential to optimizing policy effectiveness in high-dimensional deep reinforcement learning environments, thereby expanding the robustness and applicability of SAC in complex control tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。