视频生成模型通过外部反馈自我改进，提升机器人任务成功率。
VideoAgent: Self-Improving Video Generation
- 用自条件一致性方法在推理时优化视频生成质量
- 在模拟环境中任务成功率显著提升，幻觉内容大幅减少
- 适合研究机器人视觉规划与自迭代生成的学者
视频生成已用于生成控制机器人的视觉计划。以往方法在给定图像观测和语言指令后生成视频计划，并将其转换为机器人动作执行。但主要瓶颈在于生成视频质量差，常出现幻觉内容和不合理的物理行为，导致动作提取后任务成功率低。尽管扩大数据集和模型规模可部分缓解问题，但引入外部反馈对真实世界中的视频生成至关重要。为此，我们提出VideoAgent，一种基于外部反馈实现自改进的视频生成框架。不同于直接执行生成视频，VideoAgent首先通过一种新方法——自条件一致性，对视频计划进行推理时优化，将计算资源转化为更高质量的视频。在执行过程中，系统还能收集环境数据以进一步改进视频生成能力。在MetaWorld和iTHOR的模拟机器人操作实验中，VideoAgent显著减少了幻觉现象，大幅提升了下游任务的成功率。此外，我们还展示了其在真实机器人视频上的有效修正能力，初步验证了机器人作为物理世界反馈工具在视频生成地基化中的潜力。视频演示与代码见https://video-as-agent.github.io。
原文摘要 · Abstract (English)
Video generation has been used to generate visual plans for controlling robotic systems. Given an image observation and a language instruction, previous work has generated video plans which are then converted to robot controls to be executed. However, a major bottleneck in leveraging video generation for control lies in the quality of the generated videos, which often suffer from hallucinatory content and unrealistic physics, resulting in low task success when control actions are extracted from the generated videos. While scaling up dataset and model size provides a partial solution, integrating external feedback is both natural and essential for grounding video generation in the real world. With this observation, we propose VideoAgent for self-improving generated video plans based on external feedback. Instead of directly executing the generated video plan, VideoAgent first refines the generated video plans using a novel procedure which we call self-conditioning consistency, allowing inference-time compute to be turned into better generated video plans. As the refined video plan is being executed, VideoAgent can collect additional data from the environment to further improve video plan generation. Experiments in simulated robotic manipulation from MetaWorld and iTHOR show that VideoAgent drastically reduces hallucination, thereby boosting success rate of downstream manipulation tasks. We further illustrate that VideoAgent can effectively refine real-robot videos, providing an early indicator that robots can be an effective tool in grounding video generation in the physical world. Video demos and code can be found at https://video-as-agent.github.io.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。