arXiv:2410.22325cs.ROcs.AI2024-10ICLR被引 39

用机器人数据预训练视觉与动作动态,提升机械臂操作能力。

Robots Pre-train Robots: Manipulation-Centric Robotic Representation from Large-Scale Robot Datasets

  • 基于机器人数据构建视觉-动作联合表征,融合本体感知与动作信息。
  • 在4个仿真环境20个任务中性能超越基线14.8%,真实场景提升76.9%。
  • 适合追求数据高效学习和真实机器人部署的研究者。

视觉表示的预训练已显著提升机器人学习效率。由于缺乏大规模领域内机器人数据集,先前工作常利用野外人类视频进行机器人视觉表征预训练。尽管效果良好,但人类视频表征存在分布偏移问题,且缺少任务完成所必需的动作动态信息。我们首次评估了多种预训练表征与下游机器人操作任务的相关性(即‘操作中心性’)。有趣的是,发现‘操作中心性’是下游任务成功率的重要指标。基于此,我们提出操作中心表征(MCR)框架,通过捕捉视觉特征与操作过程中的动作、本体感知等动态信息,提升操作中心性。具体地,在DROID机器人数据集上预训练视觉编码器,并利用机器人本体状态与动作等运动相关数据。引入一种新型对比损失,将视觉观测与机器人的本体状态-动作动态对齐,结合行为克隆式动作预测损失及时间对比损失。在4个仿真环境20个任务上的实证结果表明,MCR相比最强基线提升14.8%。此外,在真实世界中使用UR5e机械臂进行3项任务的数据高效学习时,性能提升达76.9%。项目主页:https://robots-pretrain-robots.github.io/。

原文摘要 · Abstract (English)

The pre-training of visual representations has enhanced the efficiency of robot learning. Due to the lack of large-scale in-domain robotic datasets, prior works utilize in-the-wild human videos to pre-train robotic visual representation. Despite their promising results, representations from human videos are inevitably subject to distribution shifts and lack the dynamics information crucial for task completion. We first evaluate various pre-trained representations in terms of their correlation to the downstream robotic manipulation tasks (i.e., manipulation centricity). Interestingly, we find that the "manipulation centricity" is a strong indicator of success rates when applied to downstream tasks. Drawing from these findings, we propose Manipulation Centric Representation (MCR), a foundation representation learning framework capturing both visual features and the dynamics information such as actions and proprioceptions of manipulation tasks to improve manipulation centricity. Specifically, we pre-train a visual encoder on the DROID robotic dataset and leverage motion-relevant data such as robot proprioceptive states and actions. We introduce a novel contrastive loss that aligns visual observations with the robot's proprioceptive state-action dynamics, combined with a behavior cloning (BC)-like actor loss to predict actions during pre-training, along with a time contrastive loss. Empirical results across 4 simulation domains with 20 tasks verify that MCR outperforms the strongest baseline method by 14.8%. Moreover, MCR boosts the performance of data-efficient learning with a UR5e arm on 3 real-world tasks by 76.9%. Project website: https://robots-pretrain-robots.github.io/.

机器人学习表征预训练操作中心性多模态对齐

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。