用可微仿真训练四足机器人行走，无需复杂奖励函数即可在真实硬件上稳定运行。
Open-DiffLoco: Open-Source Differentiable Learning for Deployable Blind Quadruped Locomotion
- 基于可微仿真和短时域演员-评论家算法，实现端到端训练。
- 在真实机器狗上实现0.2米/秒以下的轨迹误差，速度超1米/秒，抗扰动强。
- 开源框架支持快速部署，单卡6GB显存20-60分钟完成训练。
通过传统强化学习开发可部署的运动策略通常需要复杂的奖励设计和高昂的训练成本。虽然可微仿真提供了高效替代方案，但支持将此类策略端到端迁移至物理硬件的开源工具仍有限。本文提出 Open-DiffLoco，一个基于可微仿真的开源框架，用于训练可部署的盲视四足运动策略。该框架在 MuJoCo XLA (MJX) 中实现短时域演员-评论家（SHAC）算法，训练仅依赖本体感受信息的策略，不依赖基底线速度观测或参考轨迹。其奖励函数大幅简化，使机器人能自主发现行走模式，无需传统方法中的辅助奖励。在实际硬件（Unitree Go2 四足机器人）上部署后，策略可实现全向速度指令跟踪，均方根误差低于0.2米/秒，最高速度超过1米/秒，并对不平地形及横向推力等外部干扰保持鲁棒性。所有配置下训练仅需单张 NVIDIA GeForce RTX 5080 GPU 的6GB以内显存，耗时约20-60分钟。作为对 SHAC 的扩展，本文提出雅可比增强价值估计（JAVE），通过监督评论家雅可比矩阵改善早期一阶策略梯度训练。据我们所知，Open-DiffLoco 是首个基于可微仿真训练可部署四足运动策略的开源框架。部署视频与源代码见：https://diffloco.martin-opat.com/
原文摘要 · Abstract (English)
Developing deployable locomotion policies through conventional reinforcement learning often requires complex reward engineering and expensive training times. While differentiable simulation offers a highly efficient alternative, open-source tools capable of end-to-end transfer of these policies to physical hardware remain limited. This paper introduces Open-DiffLoco, an open-source framework for training deployable blind quadruped locomotion policies with differentiable simulation. The framework implements the Short-Horizon Actor-Critic (SHAC) algorithm in MuJoCo XLA (MJX) and trains a proprioceptive policy that transfers to real-world hardware. The deployed policy removes privileged actor observations, including base linear velocity, and does not rely on reference trajectories. It also uses a substantially simplified reward function, enabling the robot to discover walking patterns without the complex auxiliary rewards typically used in conventional reinforcement learning pipelines. When deployed on physical hardware (a Unitree Go2 quadruped), the trained policy tracks omnidirectional velocity commands with root-mean-square error below 0.2 m/s, reaches speeds above 1 m/s, and remains robust to uneven terrain and external physical disturbances, such as lateral pushes. Across the reported configurations, training uses under 6 GB of VRAM on a single NVIDIA GeForce RTX 5080 GPU and completes in approximately 20-60 minutes. As an algorithmic extension to SHAC, we propose Jacobian-Augmented Value Estimation (JAVE), which supervises the critic Jacobians to improve early first-order policy-gradient training. To our knowledge, Open-DiffLoco is the first open-source framework for training deployable locomotion policies using differentiable simulation. Deployment videos and source code are available at: https://diffloco.martin-opat.com/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。