无需视频或深度图,仅凭一张图和指令即可预测物体6自由度运动轨迹。
DreamTraj: Generating 6-DoF Object Trajectories by Reading Unrendered Video Diffusion Latents

- 从冻结的图像到视频扩散模型的中间表示中直接读取运动信息
- 在5038条轨迹数据上实现最优的平移与旋转预测性能
- 适合需要快速、高精度运动预测的机器人抓取场景
准确预测操作过程中的物体轨迹对闭环感知-动作系统至关重要。现有研究受限于数据集缺乏细粒度语言-运动标注,以及预测模型依赖视频、深度图或CAD模型等特权输入,或通过成本高且易出错的感知管道从完全生成的视频中恢复运动。我们提出MOVE数据集,包含5,038条以物体为中心的视角轨迹,每条均配有细粒度自然语言指令而非粗粒度动词-名词标签。进一步提出DreamTraj,仅需单张RGB图像与任务指令即可预测6-DoF物体轨迹,推理时无需视频、深度图或CAD模型:不生成视频,而是读取冻结图像到视频扩散模型在早期去噪步骤中的内部表示。一个轻量级流匹配阅读器将查询-键注意力轨迹与池化隐藏状态解码为相对6-DoF位姿。据我们所知,这是首个直接从视频扩散模型中间表示中解码物体6-DoF轨迹的方法,而非生成像素。DreamTraj在平移与旋转预测上均超越使用多帧或特权输入的基准模型,且速度比生成-提取流水线快4.6倍。
原文摘要 · Abstract (English)
Accurate prediction of object trajectories during manipulation is essential for closing the perception-action loop. Progress is limited on two fronts: available datasets lack fine-grained language-to-motion annotations, and existing predictors either rely on privileged inputs such as video, depth, or CAD models, or recover motion from fully generated videos through costly, error-prone perception pipelines. We close the supervision gap with the MOVE dataset, 5,038 object-centric egocentric trajectories, each paired with a fine-grained natural-language instruction rather than a coarse verb-noun label. We further propose DreamTraj, which predicts a 6-DoF object trajectory from a single RGB image and a task instruction, requiring no video, depth, or CAD model at inference: rather than generating a video, it reads motion from the internal representations of a frozen image-to-video diffusion model at an early denoising step. A lightweight flow-matching Reader decodes query-key attention tracks and pooled hidden states into relative 6-DoF poses. To our knowledge, this is the first approach to directly decode object 6-DoF trajectories from intermediate video diffusion representations rather than generated pixels. DreamTraj sets a new state of the art on both translation and rotation against forecasters that consume multi-frame or privileged inputs, and runs 4.6x faster than generate-then-extract pipelines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。