arXiv:2505.20425cs.RO2025-05NeurIPS被引 12

用世界模型生成轨迹,让机器人仅看一次就能完成新任务。

OSVI-WM: One-Shot Visual Imitation for Unseen Tasks using World-Model-Guided Trajectory Generation

  • 通过世界模型预测隐状态和动作序列,生成可执行路径。
  • 在仿真和真实机器人上提升30%以上成功率,尤其擅长复杂新任务。
  • 适合需要快速适应新环境的机器人应用,如家庭服务或工业操作。

视觉模仿学习使机器人通过观看专家示范视频习得技能。在单次示例(one-shot)设置中,机器人仅需观察一次示范视频即可生成策略,无需额外微调。现有方法通常在相同任务集合上训练与评估,仅改变物体配置,难以泛化到语义或结构不同的未见任务。尽管部分近期方法尝试解决此问题,但在视觉相似但上下文不同、需不同响应的难题任务上仍表现不佳。此外,多数方法缺乏显式环境动态建模,限制了对未来状态的推理能力。为此,我们提出一种新型单次视觉模仿学习框架——基于世界模型引导的轨迹生成。给定专家示范视频和机器人初始观测,该方法利用学习到的世界模型预测一系列隐状态与动作。该隐状态轨迹随后被解码为物理路点,指导机器人执行。我们在两个仿真基准和三个真实机器人平台上进行评估,结果表明其持续优于先前方法,在某些情况下提升超30%。代码已开源:https://github.com/raktimgg/osvi-wm。

原文摘要 · Abstract (English)

Visual imitation learning enables robotic agents to acquire skills by observing expert demonstration videos. In the one-shot setting, the agent generates a policy after observing a single expert demonstration without additional fine-tuning. Existing approaches typically train and evaluate on the same set of tasks, varying only object configurations, and struggle to generalize to unseen tasks with different semantic or structural requirements. While some recent methods attempt to address this, they exhibit low success rates on hard test tasks that, despite being visually similar to some training tasks, differ in context and require distinct responses. Additionally, most existing methods lack an explicit model of environment dynamics, limiting their ability to reason about future states. To address these limitations, we propose a novel framework for one-shot visual imitation learning via world-model-guided trajectory generation. Given an expert demonstration video and the agent's initial observation, our method leverages a learned world model to predict a sequence of latent states and actions. This latent trajectory is then decoded into physical waypoints that guide the agent's execution. Our method is evaluated on two simulated benchmarks and three real-world robotic platforms, where it consistently outperforms prior approaches, with over 30% improvement in some cases. The code is available at https://github.com/raktimgg/osvi-wm.

视觉模仿世界模型机器人控制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。