arXiv:2506.22007cs.CV2025-06被引 17

用关键帧+插值生成机器人长时序操作视频,避免误差累积。

RoboEnvision: A Long-Horizon Video Generation Model for Multi-Task Robot Manipulation

  • 分任务生成关键帧,再用扩散模型插值生成长视频。
  • 在两个基准上视频质量和一致性均达顶尖水平。
  • 适合需要长程规划的机器人仿真与策略训练场景。

针对机器人操作任务中生成长时序视频的挑战,现有文本到视频扩散模型虽在逼真度、语言理解与运动生成上取得进展,但在长时序任务中表现不佳。当前方法通常预测单个任务的短序列并采用自回归方式扩展,导致生成视频与执行中误差累积。为此,本文提出新流程:首先将高层目标分解为原子任务并生成对齐指令的关键帧;再通过第二阶段扩散模型在关键帧间进行插值,生成长时序视频;提出语义保持注意力模块以保证关键帧间一致性;设计轻量级策略模型从生成视频中回归机器人关节状态。该方法在两个基准上实现了视频质量与一致性的最优表现,并在长时序任务中超越此前策略模型。

原文摘要 · Abstract (English)

We address the problem of generating long-horizon videos for robotic manipulation tasks. Text-to-video diffusion models have made significant progress in photorealism, language understanding, and motion generation but struggle with long-horizon robotic tasks. Recent works use video diffusion models for high-quality simulation data and predictive rollouts in robot planning. However, these works predict short sequences of the robot achieving one task and employ an autoregressive paradigm to extend to the long horizon, leading to error accumulations in the generated video and in the execution. To overcome these limitations, we propose a novel pipeline that bypasses the need for autoregressive generation. We achieve this through a threefold contribution: 1) we first decompose the high-level goals into smaller atomic tasks and generate keyframes aligned with these instructions. A second diffusion model then interpolates between each of the two generated frames, achieving the long-horizon video. 2) We propose a semantics preserving attention module to maintain consistency between the keyframes. 3) We design a lightweight policy model to regress the robot joint states from generated videos. Our approach achieves state-of-the-art results on two benchmarks in video quality and consistency while outperforming previous policy models on long-horizon tasks.

视频生成机器人扩散模型长时序

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。