用普通视频预测物体未来3D运动轨迹，无需额外信息。
MotionForesight: Re-purposing Video Models for Future 3D Scene-Flow Prediction
- 复用预训练视频模型的运动先验，转为3D场景流预测。
- 仅用4万段视频训练，跨场景泛化能力强，效果超百万视频模型。
- 适合做机器人决策、智能体交互等需要物理预判的任务。
人类能通过被动观察推断物体可能的运动：杯子可能被拿起，抽屉可能滑动，盖子可能旋转关闭。这种预测揭示了真实世界中行动所需的物理后果。本文研究如何从普通单目人物-物体交互视频中学习此类预见能力。给定一段短时观察视频，MotionForesight 预测被操作物体上点的未来3D轨迹。这将交互预测转化为不依赖物体属性的对象中心3D运动预报。核心洞察是：视频预测模型已蕴含丰富的物体在人机交互中的运动先验。我们将其从像素预测转向未来3D场景流预测。基于预训练视频模型构建密集3D跟踪器，从完整视频片段生成伪真值轨迹，仅使用观察帧训练预测器。用学习到的掩码隐变量替代未来RGB与几何信息，训练轻量适配器将回溯性跟踪表示转化为前向预测，同时冻结大型视频与跟踪组件。仅使用4万段人类视频，且无需语言等辅助输入，MotionForesight在多样化的分布外物体、环境、视角和交互上均表现良好，显著优于使用超百万视频训练的大模型。结果表明，可高效重用视频先验，生成用于具身智能的显式几何预测。
原文摘要 · Abstract (English)
Humans can infer how objects are likely to move from passive observation: a cup may be lifted, a drawer may slide, and a lid may rotate shut. Such predictions expose the physical consequences of interaction needed to act in the real world. We study how to learn this anticipation from ordinary monocular videos of human-object interaction. Given a short observed video context, MotionForesight predicts future 3D trajectories for points on the manipulated object. This casts interaction prediction as object-centered 3D motion forecasting without any assumptions on the object properties. Our key insight is that video prediction models already encode rich priors about how objects move during human interactions. We redirect these priors from pixel prediction toward future 3D scene flow. We start from a dense 3D tracker built on a pretrained video model, generate pseudo-ground-truth tracks from complete clips, and train the forecaster using only the observed frames. We replace future RGB and geometry with learned mask latents and train a lightweight adapter to turn the retrospective tracking representation into a forward predictor, while freezing the large video and tracking components. Using just 40k human videos and no auxiliary inputs such as language, MotionForesight generalizes across diverse out-of-distribution objects, environments, viewpoints, and interactions. It also outperforms substantially larger models that use over a million training videos. These results show that we can efficiently re-purpose video priors into explicit geometric forecasts for embodied intelligence. https://motionforesight.github.io/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。