用大规模视频预训练构建通用机器人规划模型,实现零样本任务泛化。
Large Video Planner Enables Generalizable Robot Control
- 基于互联网规模视频数据训练生成式机器人规划模型
- 在真实场景中成功执行新任务,零样本泛化效果显著
- 适合希望实现通用机器人控制的研究者与开发者
通用机器人需要能在多样化任务和环境中泛化的决策模型。现有方法通过将多模态大语言模型(MLLM)扩展为输出动作的系统,构建视觉-语言-动作(VLA)模型,依赖其大规模语言与图像预训练能力迁移到动作输出。本文探索一种新范式:以大规模视频预训练为核心构建机器人基础模型。相比静态图像和语言,视频能自然捕捉物理世界中的时空状态与动作序列,更契合机器人行为。我们收集了互联网规模的人类活动与任务演示视频数据集,并首次在基础模型级别训练了一个面向生成式机器人规划的开放视频模型。该模型可对新场景与任务生成零样本视频计划,经后处理提取可执行机器人动作。通过第三方选定的真实世界任务及实机实验评估,验证了模型在任务级泛化上的有效性,实现了成功物理执行。结果表明该模型具备强指令跟随、良好泛化性和现实可行性。我们已公开模型与数据集,支持可复现的视频驱动机器人学习。官网:https://www.boyuan.space/large-video-planner/
原文摘要 · Abstract (English)
General-purpose robots require decision-making models that generalize across diverse tasks and environments. Recent works build robot foundation models by extending multimodal large language models (MLLMs) with action outputs, creating vision-language-action (VLA) systems. These efforts are motivated by the intuition that MLLMs' large-scale language and image pretraining can be effectively transferred to the action output modality. In this work, we explore an alternative paradigm of using large-scale video pretraining as a primary modality for building robot foundation models. Unlike static images and language, videos capture spatio-temporal sequences of states and actions in the physical world that are naturally aligned with robotic behavior. We curate an internet-scale video dataset of human activities and task demonstrations, and train, for the first time at a foundation-model scale, an open video model for generative robotics planning. The model produces zero-shot video plans for novel scenes and tasks, which we post-process to extract executable robot actions. We evaluate task-level generalization through third-party selected tasks in the wild and real-robot experiments, demonstrating successful physical execution. Together, these results show robust instruction following, strong generalization, and real-world feasibility. We release both the model and dataset to support open, reproducible video-based robot learning. Our website is available at https://www.boyuan.space/large-video-planner/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。