arXiv:2608.13489cs.CVcs.RO2026-08被引 1

让机器人预测动作后的视觉变化,精准保持物体和机械臂状态。

DreamX-Phi 1.0: Action-Conditioned Video World Model for Robotic Manipulation

论文配图:DreamX-Phi 1.0: Action-Conditioned Video World Model for Robotic Manipulation
图 1 · 摘自论文原文
  • 用几何编码确保每只机械臂运动路径正确
  • 加入深度分支和掩码机制维持物体一致性
  • 模型轻量化后仍保持高精度,适合实际部署

我们提出 DreamX-Phi 1.0,一种用于机器人操作的动作条件视频世界模型。给定观测帧、语言指令和预设的动作序列(包含末端执行器位姿与夹爪状态),模型可预测未来视觉结果。仅追求真实感不足以保证准确性:逼真的生成可能错误移动某只机械臂或丢失目标物体。为此,我们通过类似 PRoPE 的几何编码,将每个机械臂的 $ ext{SE}(3)$ 变换注入注意力机制,保留机械臂身份与刚体运动结构。单靠动作控制无法完全约束场景几何或小物体演化,因此引入轻量级深度分支以建模场景级几何,并采用冻结的 V-JEPA 教师模型生成 SAM3 掩码,确保抓取过程中物体一致性。此外,通过分布匹配蒸馏将多步生成器压缩为少数几步学生模型,提升部署效率。截至撰写时,该模型在 WorldArena 2.0 Challenge Track 1 中排名第一,Track 2 中排名第二。模型与代码将公开可用。

原文摘要 · Abstract (English)

We present \textbf{DreamX-Phi 1.0}, an action-conditioned video world model for robotic manipulation that, given an observed frame, a language instruction, and a prescribed action sequence comprising end-effector poses and gripper states, predicts the resulting future observations. Yet realism alone does not guarantee faithfulness: a convincing rollout can still move the wrong arm or lose the manipulated object. To ensure the prediction respects each arm's commanded path, we inject per-arm $\mathrm{SE}(3)$ transformations into attention via \textbf{PRoPE-style geometric encoding}, preserving arm identity and rigid-motion structure. Action control alone does not fully constrain scene geometry or the evolution of small manipulated objects. We therefore add a lightweight \textbf{depth branch} for scene-level geometry and use \textbf{SAM3 masks} with a frozen \textbf{V-JEPA teacher} to maintain object consistency throughout grasping. We further distill the multi-step generator into a few-step student via distribution-matching distillation for efficient deployment. At the time of writing, \model{} achieves first place on Track~1 and second place on Track~2 of the WorldArena~2.0 Challenge. Our model and code will be publicly available.

机器人操作世界模型视频生成动作预测

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。