让机器人在新设备和视角下也能零样本迁移，靠的是3D场景自适应编码器。
Adapt3R: Adaptive 3D Scene Representation for Domain Transfer in Imitation Learning
- 用预训练2D模型提取语义，3D仅用于定位末端执行器位置。
- 93个仿真+6个真实任务中，实现零样本跨设备、跨视角迁移。
- 可适配任意模仿学习算法，保持原模型学习能力。
模仿学习能训练机器人完成复杂多样的操作任务，但其策略在训练分布外的观测下表现脆弱。已有研究提出利用校准的RGBD相机观测构建3D场景表示以缓解此问题，但在未见机器人本体和相机视角下的评估中，效果仅略有提升。为此，我们提出Adapt3R——一种通用的3D观测编码器，能将校准的RGBD数据合成向量，作为任意模仿学习算法的条件输入。核心思想是使用预训练2D主干网络提取语义信息，仅用3D作为定位语义相对于末端执行器位置的媒介。我们在93个仿真任务和6个真实任务中验证，当与多种模仿学习算法端到端联合训练时，Adapt3R在保持原算法学习能力的同时，实现了对新本体和新相机姿态的零样本迁移。
原文摘要 · Abstract (English)
Imitation Learning can train robots to perform complex and diverse manipulation tasks, but learned policies are brittle with observations outside of the training distribution. 3D scene representations that incorporate observations from calibrated RGBD cameras have been proposed as a way to mitigate this, but in our evaluations with unseen embodiments and camera viewpoints they show only modest improvement. To address those challenges, we propose Adapt3R, a general-purpose 3D observation encoder which synthesizes data from calibrated RGBD cameras into a vector that can be used as conditioning for arbitrary IL algorithms. The key idea is to use a pretrained 2D backbone to extract semantic information, using 3D only as a medium to localize this information with respect to the end-effector. We show across 93 simulated and 6 real tasks that when trained end-to-end with a variety of IL algorithms, Adapt3R maintains these algorithms' learning capacity while enabling zero-shot transfer to novel embodiments and camera poses.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。