用扩散模型提升四足机器人运动控制，仅靠行为差异就能自动优化动作。
Preference Aligned Diffusion Planner for Quadrupedal Locomotion Control
- 分两阶段训练：先离线学专家数据中的状态-动作分布，再在线模拟中扩充行为多样性。
- 在不同速度下稳定实现快走、小跑、跳跃步态，零样本迁移到真实机器人。
- 无需奖励标签或人工偏好，仅靠行为对比就能有效优化，适合强化学习初学者。
扩散模型在捕捉大规模数据中的复杂分布方面表现优异，为四足机器人运动控制提供了有前景的解决方案。然而，扩散规划器的鲁棒性依赖于预采集数据集的多样性。为缓解此问题，我们提出一种两阶段学习框架，在数据有限的情况下增强扩散规划器的能力（无奖励依赖）。在离线阶段，扩散规划器从专家数据集中学习状态-动作序列的联合分布，无需使用奖励标签。随后，在仿真环境中基于已训练的离线规划器进行在线交互，显著丰富了原始行为，从而提升了鲁棒性。具体地，我们提出一种新颖的弱偏好标注方法，无需真实奖励或人类偏好。该方法在不同速度下均表现出更优的稳定性与速度跟踪精度，适用于快走、小跑和跳跃步态，并可实现零样本迁移至真实的 Unitree Go1 机器人。项目主页见：https://shangjaven.github.io/preference-aligned-diffusion-legged。
原文摘要 · Abstract (English)
Diffusion models demonstrate superior performance in capturing complex distributions from large-scale datasets, providing a promising solution for quadrupedal locomotion control. However, the robustness of the diffusion planner is inherently dependent on the diversity of the pre-collected datasets. To mitigate this issue, we propose a two-stage learning framework to enhance the capability of the diffusion planner under limited dataset (reward-agnostic). Through the offline stage, the diffusion planner learns the joint distribution of state-action sequences from expert datasets without using reward labels. Subsequently, we perform the online interaction in the simulation environment based on the trained offline planner, which significantly diversified the original behavior and thus improves the robustness. Specifically, we propose a novel weak preference labeling method without the ground-truth reward or human preferences. The proposed method exhibits superior stability and velocity tracking accuracy in pacing, trotting, and bounding gait under different speeds and can perform a zero-shot transfer to the real Unitree Go1 robots. The project website for this paper is at https://shangjaven.github.io/preference-aligned-diffusion-legged.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。