提出MMP模型，实现动态场景下多帧单目几何的前馈估计。
Learning Multi-frame and Monocular Prior for Estimating Geometry in Dynamic Scenes
- 基于孪生网络设计轨迹编码模块，捕捉点级动态变化。
- 在多帧上实现前馈点云生成，相比基线降低15.1%回归误差。
- 适合需要实时、高精度动态3D重建的应用场景。
在单目视频捕捉动态场景时，估计视频内容的三维几何结构是计算机视觉中的基础挑战。由于物体运动的影响，现有模型通常仅能预测部分属性，如仅限于两帧之间的深度图或点云图。这些属性在多帧下固有噪声大，常需测试时全局优化来恢复完整几何结构，但该方法易失败且推理开销高。为此，本文提出新模型MMP，以前馈方式估计动态场景的几何信息，生成随时间演化的动态点云表示。基于最新孪生架构，引入新的轨迹编码模块，将每帧点级动态映射到特征表示中，显著增强对动态场景的表达能力。实验表明，MMP在前馈点云预测任务中达到当前最优性能，回归误差降低15.1%。
原文摘要 · Abstract (English)
In monocular videos that capture dynamic scenes, estimating the 3D geometry of video contents has been a fundamental challenge in computer vision. Specifically, the task is significantly challenged by the object motion, where existing models are limited to predict only partial attributes of the dynamic scenes, such as depth or pointmaps spanning only over a pair of frames. Since these attributes are inherently noisy under multiple frames, test-time global optimizations are often employed to fully recover the geometry, which is liable to failure and incurs heavy inference costs. To address the challenge, we present a new model, coined MMP, to estimate the geometry in a feed-forward manner, which produces a dynamic pointmap representation that evolves over multiple frames. Specifically, based on the recent Siamese architecture, we introduce a new trajectory encoding module to project point-wise dynamics on the representation for each frame, which can provide significantly improved expressiveness for dynamic scenes. In our experiments, we find MMP can achieve state-of-the-art quality in feed-forward pointmap prediction, e.g., 15.1% enhancement in the regression error.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。