用视频和自然语言指令生成机器人操作代码，实现零样本泛化
Robotic Programmer: Video Instructed Policy Code Generation for Robotic Manipulation
- 通过视觉-语言模型从真实世界视频中合成可执行代码
- 在RLBench上零样本成功率比GPT-4o高11.6%，接近有监督基线
- 支持多种机器人、任务和技能接口，适合实际部署场景
跨机器人、任务和环境的零样本泛化仍是机器人操作中的重大挑战。策略代码生成方法通过可执行代码连接高层任务描述与底层动作序列，利用大语言模型和原子技能库的泛化能力。本文提出机器人基础模型Robotic Programmer（RoboPro），具备感知视觉信息并遵循自由形式指令执行机器人操作的能力，可在零样本条件下生成策略代码。为解决获取运行时代码数据效率低、成本高的问题，我们设计Video2Code，利用现成的视觉-语言模型与代码域大语言模型，从大量真实视频中合成可执行代码。大量实验表明，RoboPro在模拟器和真实环境中均达到当前最优的零样本性能。具体而言，在RLBench上的零样本成功率达87.3%，比当前最优模型GPT-4o高出11.6%，甚至接近强监督训练基线。此外，RoboPro对API格式和技能集变化具有鲁棒性。
原文摘要 · Abstract (English)
Zero-shot generalization across various robots, tasks and environments remains a significant challenge in robotic manipulation. Policy code generation methods use executable code to connect high-level task descriptions and low-level action sequences, leveraging the generalization capabilities of large language models and atomic skill libraries. In this work, we propose Robotic Programmer (RoboPro), a robotic foundation model, enabling the capability of perceiving visual information and following free-form instructions to perform robotic manipulation with policy code in a zero-shot manner. To address low efficiency and high cost in collecting runtime code data for robotic tasks, we devise Video2Code to synthesize executable code from extensive videos in-the-wild with off-the-shelf vision-language model and code-domain large language model. Extensive experiments show that RoboPro achieves the state-of-the-art zero-shot performance on robotic manipulation in both simulators and real-world environments. Specifically, the zero-shot success rate of RoboPro on RLBench surpasses the state-of-the-art model GPT-4o by 11.6%, which is even comparable to a strong supervised training baseline. Furthermore, RoboPro is robust to variations on API formats and skill sets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。