arXiv:2410.15959cs.ROcs.CV2024-10被引 43

用扩散Transformer直接生成连续动作序列,提升机器人泛化能力。

Diffusion Transformer Policy

  • 用大模型扩散Transformer直接去噪动作块,替代小头预测离散动作。
  • 在Calvin ABC->D任务中单视角下连续完成任务数达3.6,优于现有方法。
  • 适合需要复杂连续动作控制的机器人任务,尤其看重泛化能力的研究者。

近期基于多样化机器人数据集预训练的大规模视觉-语言-动作模型展示了在少量领域内数据下泛化到新环境的潜力。然而,这些方法通常通过小型动作头预测离散或连续动作,限制了对多样化动作空间的处理能力。本文提出扩散Transformer策略(Diffusion Transformer Policy),采用大型多模态扩散Transformer建模连续动作序列,直接通过大型Transformer模型对动作块进行去噪,而非使用小型动作头生成动作嵌入。借助Transformer的可扩展性,该方法能有效建模跨大规模多样化机器人数据集的连续末端执行器动作,并实现更优的泛化性能。大量实验表明,Diffusion Transformer Policy在Maniskill2、Libero、Calvin和SimplerEnv上均表现优异,且在Real-to-Sim基准的SimplerEnv及真实世界Franka机械臂、Libero任务上持续优于OpenVLA和Octo。无需额外技巧,在Calvin任务ABC->D中仅使用单个第三视角摄像头,平均连续完成任务数达到3.6,预训练阶段使成功序列长度提升超过1.2。

原文摘要 · Abstract (English)

Recent large vision-language-action models pretrained on diverse robot datasets have demonstrated the potential for generalizing to new environments with a few in-domain data. However, those approaches usually predict individual discretized or continuous action by a small action head, which limits the ability in handling diverse action spaces. In contrast, we model the continuous action sequence with a large multi-modal diffusion transformer, dubbed as Diffusion Transformer Policy, in which we directly denoise action chunks by a large transformer model rather than a small action head for action embedding. By leveraging the scaling capability of transformers, the proposed approach can effectively model continuous end-effector actions across large diverse robot datasets, and achieve better generalization performance. Extensive experiments demonstrate the effectiveness and generalization of Diffusion Transformer Policy on Maniskill2, Libero, Calvin and SimplerEnv, as well as the real-world Franka arm, achieving consistent better performance on Real-to-Sim benchmark SimplerEnv, real-world Franka Arm and Libero compared to OpenVLA and Octo. Specifically, without bells and whistles, the proposed approach achieves state-of-the-art performance with only a single third-view camera stream in the Calvin task ABC->D, improving the average number of tasks completed in a row of 5 to 3.6, and the pretraining stage significantly facilitates the success sequence length on the Calvin by over 1.2.

机器人扩散模型动作生成视觉-语言

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。