让机器人视频生成更真实，跨视角保持3D几何一致
Geometry-aware 4D Video Generation for Robot Manipulation
- 用多视角点云对齐监督，强制生成视频的3D一致性
- 仅需单张RGB-D图，就能从新视角生成时空连贯视频
- 生成的4D视频可直接用于机器人抓取轨迹预测，泛化性强
理解物理世界的动态变化能提升机器人在复杂环境中的规划与交互能力。尽管近期视频生成模型在建模动态场景方面展现出强大潜力，但生成在时间上连贯且跨视角几何一致的视频仍是重大挑战。为此，我们提出一种4D视频生成模型，通过训练时引入跨视角点云对齐作为监督信号，强制模型学习统一的3D场景表征。借助这种几何监督，模型可仅凭每视角一张RGB-D图像，无需相机位姿输入，便能从新视角生成时空对齐的未来视频序列。相比现有基线方法，本方法在多个模拟与真实机器人数据集上均生成了更视觉稳定、空间对齐的预测结果。进一步实验表明，生成的4D视频可结合现成的6DoF位姿追踪器恢复机器人末端执行器轨迹，从而获得在新视角下具有良好泛化能力的机器人操作策略。
原文摘要 · Abstract (English)
Understanding and predicting dynamics of the physical world can enhance a robot's ability to plan and interact effectively in complex environments. While recent video generation models have shown strong potential in modeling dynamic scenes, generating videos that are both temporally coherent and geometrically consistent across camera views remains a significant challenge. To address this, we propose a 4D video generation model that enforces multi-view 3D consistency of generated videos by supervising the model with cross-view pointmap alignment during training. Through this geometric supervision, the model learns a shared 3D scene representation, enabling it to generate spatio-temporally aligned future video sequences from novel viewpoints given a single RGB-D image per view, and without relying on camera poses as input. Compared to existing baselines, our method produces more visually stable and spatially aligned predictions across multiple simulated and real-world robotic datasets. We further show that the predicted 4D videos can be used to recover robot end-effector trajectories using an off-the-shelf 6DoF pose tracker, yielding robot manipulation policies that generalize well to novel camera viewpoints.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。