arXiv:2603.05757cs.RO2026-03被引 1

用语言模型约束视频生成,让机器人零样本操作更精准。

EmboAlign: Aligning Video Generation with Compositional Constraints for Zero-Shot Manipulation

  • 用视觉语言模型提取任务约束,指导视频生成与轨迹优化。
  • 在6个真实机器人任务中提升成功率43.3个百分点。
  • 无需额外训练数据,适合复杂动作的零样本操控场景。

视频生成模型(VGMs)虽能生成连贯的物体动态视频,但常产生物理上不合理的运动。通过几何重定向将像素空间运动转为机械臂动作时,深度估计与关键点追踪的误差会累积。为此,我们提出EmboAlign,一种无需训练数据的推理阶段对齐框架。该方法利用视觉语言模型(VLMs)在推理时生成任务相关的组合性约束,弥补VGM缺乏结构化空间推理的不足。给定语言指令后,VLM自动提取任务特定的物理约束,并在两个阶段应用:(1)约束引导的视频回放筛选,从一批生成视频中选出最符合物理规律的候选;(2)基于约束的轨迹优化,以选中的视频为初始轨迹,进一步修正重定向误差。在六个需精确执行的现实机器人操纵任务上评估,相较最强基线成功率达提升43.3%点,且无需任何任务专属训练数据。

原文摘要 · Abstract (English)

Video generative models (VGMs) pretrained on large-scale internet data can produce temporally coherent rollout videos that capture rich object dynamics, offering a compelling foundation for zero-shot robotic manipulation. However, VGMs often produce physically implausible rollouts, and converting their pixel-space motion into robot actions through geometric retargeting further introduces cumulative errors from imperfect depth estimation and keypoint tracking. To address these challenges, we present \method{}, a data-free framework that aligns VGM outputs with compositional constraints generated by vision-language models (VLMs) at inference time. The key insight is that VLMs offer a capability complementary to VGMs: structured spatial reasoning that can identify the physical constraints critical to the success and safety of manipulation execution. Given a language instruction, \method{} uses a VLM to automatically extract a set of compositional constraints capturing task-specific requirements, which are then applied at two stages: (1) constraint-guided rollout selection, which scores and filters a batch of VGM rollouts to retain the most physically plausible candidate, and (2) constraint-based trajectory optimization, which uses the selected rollout as initialization and refines the robot trajectory under the same constraint set to correct retargeting errors. We evaluate \method{} on six real-robot manipulation tasks requiring precise, constraint-sensitive execution, improving the overall success rate by 43.3\% points over the strongest baseline without any task-specific training data.

视频生成机器人操控零样本约束对齐

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。