用简单方法直接估计动态场景的几何结构,无需复杂流程。
MonST3R: A Simple Approach for Estimating Geometry in the Presence of Motion

- 以逐帧点云方式直接建模动态场景几何
- 在有限动态数据上微调,仍实现良好深度与位姿估计
- 适合需要高效4D重建的视频应用
从动态场景中估计几何结构仍是计算机视觉的核心挑战。现有方法多依赖多阶段流水线或全局优化,将问题分解为深度与光流等子任务,导致系统复杂且易出错。本文提出运动-杜斯3R(MonST3R),一种几何优先的新方法,可直接从动态场景中逐时刻估计几何结构。核心思想是仅对每个时间步估计点图,即可有效适配原本仅用于静态场景的DUST3R表示。然而该方法面临训练数据稀缺的难题——缺乏带深度标注的动态、摆拍视频。尽管如此,我们通过将其视为微调任务,识别多个适用数据集,并策略性地在有限数据上训练模型,意外发现模型能处理动态,甚至无需显式运动表示。基于此,我们为若干下游视频任务引入新优化,展示了在视频深度与相机位姿估计上的优异性能,优于先前工作,在鲁棒性和效率方面表现突出。此外,MonST3R在主要前馈式4D重建任务中也展现出良好潜力。
原文摘要 · Abstract (English)
Estimating geometry from dynamic scenes, where objects move and deform over time, remains a core challenge in computer vision. Current approaches often rely on multi-stage pipelines or global optimizations that decompose the problem into subtasks, like depth and flow, leading to complex systems prone to errors. In this paper, we present Motion DUSt3R (MonST3R), a novel geometry-first approach that directly estimates per-timestep geometry from dynamic scenes. Our key insight is that by simply estimating a pointmap for each timestep, we can effectively adapt DUST3R's representation, previously only used for static scenes, to dynamic scenes. However, this approach presents a significant challenge: the scarcity of suitable training data, namely dynamic, posed videos with depth labels. Despite this, we show that by posing the problem as a fine-tuning task, identifying several suitable datasets, and strategically training the model on this limited data, we can surprisingly enable the model to handle dynamics, even without an explicit motion representation. Based on this, we introduce new optimizations for several downstream video-specific tasks and demonstrate strong performance on video depth and camera pose estimation, outperforming prior work in terms of robustness and efficiency. Moreover, MonST3R shows promising results for primarily feed-forward 4D reconstruction.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。