arXiv:2607.22535cs.ROcs.CV2026-07被引 2

将机器人动作分解为轨迹与渲染几何,提升视觉世界模型的泛化能力。

Robot-Factored World Models via Robot Rendering

论文配图:Robot-Factored World Models via Robot Rendering
图 1 · 摘自论文原文
  • 用机器人自身控制器生成可部署的轨迹,避免学习动作实现过程。
  • 通过URDF渲染机器人几何,分离机器人外观与场景交互建模。
  • 支持未见过的机器人形态推理,可从人类演示生成机器人操作视频。

动作条件视频世界模型从初始观测和动作信号预测未来观测。在机器人任务中,动作通过两个过程影响未来:首先由机器人本体与控制器实现为运动,随后场景通过接触与物体运动响应。直接对动作命令建模会要求世界模型自行学习动作实现,而对已记录未来状态建模则会泄露本应预测的交互结果。本文提出机器人因子化世界模型,将两个机器人特有因素移出世界模型。第一,动作实现:每个命令经由机器人控制器与运动学计算生成可部署的名义轨迹,作为中间信号,避免动作实现学习与未来状态泄露。第二,机器人渲染:该名义轨迹通过机器人URDF进行渲染,将机器人的几何、运动学与外观从模型中解耦,显式表示为渲染的机器人几何。为解决深度模糊问题,将末端执行器深度与场景深度配对,提供接触与遮挡的几何线索,超越图像平面重叠。相机感知的静态RGB/深度上下文与渲染机器人几何共同构成共享的视觉世界模型接口,保持视角与机器人形态间的稳定性,使模型仅通过可见的机器人几何感知动作,并学习物体如何响应。实验表明,该渲染接口优于向量条件基线,并能在推理时泛化至未见机器人形态。进一步证明,模型可通过重定向并渲染手部运动为机器人几何,从人类演示生成机器人操作视频。

原文摘要 · Abstract (English)

Action-conditioned video world models predict future observations from an initial observation and an action signal. In robotics, actions influence future observations through two distinct processes: they are first realized into robot motion by the robot body and controller, and the scene then responds through contact and object motion. Conditioning directly on action commands asks the world model to learn the realization process itself, while conditioning on logged future states leaks the interaction outcomes it is meant to predict. We propose robot-factored world models, which move two robot-specific factors outside the world model. First, action realization: each command is rolled through the robot's own controller and kinematics into a deployment-available nominal trajectory, a middle signal that avoids both action-realization learning and future-state leakage. Second, robot rendering: this nominal trajectory is rendered through the robot URDF, factoring the robot's geometry, kinematics, and appearance out of the model and into explicit rendered robot geometry. To resolve depth ambiguity, we pair end-effector depth with scene depth, giving geometric cues for contact and occlusion beyond image-plane overlap. Together, camera-aware static RGB/depth context and rendered robot geometry form a shared visual world-model interface that stays consistent across viewpoints and robot embodiments, so the model sees the action only as visible robot geometry and learns how objects respond to it. Our experiments show that the rendered interface outperforms vector-conditioned baselines and generalizes to unseen robot embodiments at inference. We further demonstrate that our model generates robot manipulation videos from human demonstrations by retargeting and rendering the hand motion as robot geometry.

世界模型机器人渲染动作泛化视觉预测

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。