提出FastDSAC,让机器人走路更稳更快
FastDSAC: Enhancing Policy Plasticity via Constrained Exploration for Scalable Humanoid Locomotion

- 用截断高斯分布约束动作,减少无效探索
- 训练速度比现有方法快,收敛更快
- 适合需要快速迭代的复杂机器人任务
可扩展强化学习推动了高吞吐采样架构的发展,显著缩短了离策略方法在机器人行走任务中的训练时间。然而,数据量和更新频率的激增会破坏基于价值的方法稳定性,并降低策略网络的可塑性。为此,本文提出FastDSAC,一种专为并行采样设计的分布演员-评论家算法高效变体。我们引入截断高斯分布近似学习到的策略,有效排除导致目标值估计压力的分布外动作,同时保留必要的随机性以支持探索。该动作约束作为隐式正则化,抵消了剧烈梯度更新通常引起的可塑性损失。这种网络适应能力的保持提升了样本效率,尤其在高更新/数据比场景下,加速了早期训练过程。与依赖离散值分布的以往快速强化学习方法不同,本方法采用带有自适应方差调节的连续高斯表示,通过采样高置信度、高信息量的转移样本,提升价值估计精度。在MuJoCo Playground和HumanoidBench上的大量实验表明,FastDSAC不仅稳定了整体训练过程,还实现了优于当前最先进基线的渐近性能和更快的收敛速度。
原文摘要 · Abstract (English)
Scalable reinforcement learning has popularized high-throughput sampling architectures, which significantly compresses the training time for off-policy methods in robotic locomotion. However, the rapid increase of data volume and update frequency undermines the stability of value-based methods and diminishes the plasticity of policy networks. To address these challenges, this work presents FastDSAC, a fast and high-performance variant of the Distributional Actor-Critic algorithm designed for parallel sampling scenarios. Specifically, we introduce a truncated Gaussian distribution to approximate the learned policy, which effectively excludes out-of-distribution actions that strain target value estimation while keeping necessary stochasticity for exploration. The proposed action constraint functions as an implicit regularization, which counteracts the plasticity loss typically caused by aggressive gradient updates. This preservation of network adaptability enhances sample efficiency, particularly in scenarios with a high update-to-data ratio, and accelerates the early training process. In contrast to prior fast reinforcement learning approaches that rely on discrete value distributions, our method utilizes a continuous Gaussian representation equipped with adaptive variance regulation, which improves value estimation accuracy by sampling confident and informative transitions. Extensive experiments on MuJoCo Playground and HumanoidBench demonstrate that FastDSAC not only stabilizes the overall training process but also achieves superior asymptotic performance and faster convergence compared to state-of-the-art baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。