让SAC在大规模仿真中达到PPO的足式机器人运动性能
Bridging the Gap: Enabling Soft Actor Critic for High Performance Legged Locomotion
- 改进SAC的策略初始化与目标估计机制
- 在多平台任务上实现与PPO相当的训练效果
- 适合需要仿真到现实迁移的机器人控制研究者
近端策略优化（PPO）因在大规模并行仿真环境（如IsaacLab）中表现出强鲁棒性和可扩展性，已成为足式机器人训练的主流方法。然而其在线策略特性导致样本效率低，难以用于真实硬件上的连续适应与微调。相比之下，软动作评价（SAC）是离线策略算法，可重用历史经验，更适合从仿真到现实的迁移流程。尽管如此，SAC在大规模并行训练中始终未能达到PPO的实证性能。本文识别出这一差距的根本原因，并引入三项针对性改进：策略初始化优化、考虑超时的批评者目标设计以及多步回报估计。这些改进使SAC能在大规模环境下稳定训练。在多个足式机器人平台和多样化运动任务上评估表明，所提方法完全弥合了与PPO的性能差距。
原文摘要 · Abstract (English)
Proximal Policy Optimization (PPO) has become the de facto standard for training legged robots, thanks to its robustness and scalability in massively parallel simulation environments like IsaacLab. However, its on-policy nature makes it inherently sample-inefficient, preventing its use for continuous adaptation and fine-tuning on real hardware. Soft Actor-Critic (SAC), by contrast, is an off-policy algorithm that can reuse past experience, making it a natural candidate for sim-to-real transfer workflows where the same algorithm can be used both in simulation and for online learning on the real robot. Despite these advantages, SAC has consistently failed to match PPO's empirical performance in massively parallel training settings. This work identifies the root causes of this gap and introduces targeted modifications, covering policy initialization, timeout-aware critic targets, and multi-step return estimation, that enable SAC to train stably at scale. Evaluated across multiple legged robot platforms and diverse locomotion tasks, our approach closes the performance gap with PPO entirely.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。