arXiv:2411.00508cs.RO2024-11中稿 · RSS 2025被引 22

用自然语言教机器人干活,效果好且模型更轻。

CLIP-RT: Learning Language-Conditioned Robotic Policies from Natural Language Supervision

  • 用自然语言指令收集机器人演示数据并增强。
  • 在真实场景中成功率达76%,比70亿参数模型高24%。
  • 适合非专家快速训练机器人,支持少样本泛化和人机协作。

在现实环境中教会机器人技能仍具挑战性,尤其对非专业人士而言。核心瓶颈在于采集机器人数据通常需要专业知识或专用硬件,限制了可及性和可扩展性。本文认为自然语言提供了一种直观且易用的机器人学习接口。为此,我们研究两个方面:(1) 让非专业人士通过自然语言监督(如“把手臂往右移动”)收集机器人数据;(2) 直接从这些语言监督中训练机器人策略。我们提出一种数据收集框架,基于自然语言指令生成并增强机器人示范数据。随后,提出CLIP-RT,一种新的视觉-语言-动作(VLA)模型,可从这类语言监督中学习语言条件的视觉运动策略。CLIP-RT基于预训练的CLIP模型,通过对比模仿学习预测基于语言的动作基元。我们在Open X-Embodiment数据集上训练CLIP-RT,并在本框架收集的领域内数据上微调。在真实世界评估中,CLIP-RT展现出强大学习新操作技能的能力,平均成功率相比70亿参数的OpenVLA提升24%,同时仅使用10亿参数(7倍更少)。我们进一步评估其在少样本泛化和与大模型或人类协作场景中的能力。在模拟环境中,CLIP-RT在LIBERO基准上达到93.1%的平均成功率,推理吞吐量为163 Hz。

原文摘要 · Abstract (English)

Teaching robots desired skills in real-world environments remains challenging, especially for non-experts. A key bottleneck is that collecting robotic data often requires expertise or specialized hardware, limiting accessibility and scalability. We posit that natural language offers an intuitive and accessible interface for robot learning. To this end, we study two aspects: (1) enabling non-experts to collect robotic data through natural language supervision (e.g., "move the arm to the right") and (2) training robot policies directly from this supervision. Specifically, we introduce a data collection framework that collects robot demonstrations based on natural language supervision and further augments these demonstrations. We then present CLIP-RT, a new vision-language-action (VLA) model that learns language-conditioned visuomotor policies from this supervision. CLIP-RT adapts the pretrained CLIP model and learns to predict language-based motion primitives via contrastive imitation learning. We train CLIP-RT on the Open X-Embodiment dataset and finetune it on in-domain data collected by our framework. In real-world evaluations, CLIP-RT demonstrates strong capabilities in learning novel manipulation skills, outperforming OpenVLA (7B parameters) by 24% in average success rates, while using 7x fewer parameters (1B). We further assess CLIP-RT's capabilities in few-shot generalization and collaborative scenarios involving large pretrained models or humans. In simulated environments, CLIP-RT also yields strong performance, achieving a 93.1% average success rate on the LIBERO benchmark with an inference throughput of 163 Hz.

机器人学习自然语言控制少样本泛化视觉-语言-动作

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。