多视角图像掩码建模，让模型更好理解3D几何结构。
MuM: Multi-View Masked Image Modeling for 3D Vision
- 用多视角掩码自编码，统一处理任意数量视角图像。
- 在重建、匹配和位姿估计任务上超越DINOv3和CroCo v2。
- 结构简单可扩展，适合需要几何推理的3D视觉任务。
图像自监督学习旨在从无标注数据中提取有意义的视觉表征。当扩展到大规模数据集时，该范式已达到顶尖性能，如DINOv3等模型被广泛采用。然而，多数方法侧重于语义理解而非几何推理。一个例外是跨视角补全（CroCo），一种针对3D理解设计的掩码自编码（MAE）方法。本文延续此方向，提出多视角掩码图像建模（MuM），将MAE推广至任意数量场景视角。通过统一掩码所有视角并使用带帧间注意力的轻量解码器，该方法比CroCo更简洁且更具可扩展性。我们在下游任务如前馈重建、密集图像匹配和相对位姿估计上对模型进行评估，结果表明其性能优于当前最优视觉编码器DINOv3和CroCo v2。
原文摘要 · Abstract (English)
Self-supervised learning on images seeks to extract meaningful visual representations from unlabeled data. When scaled to large datasets, this paradigm has achieved state-of-the-art performance and the resulting trained models such as DINOv3 have seen widespread adoption. However, most prior efforts are optimized for semantic understanding rather than geometric reasoning. One important exception is Cross-View Completion, CroCo, which is a form of masked autoencoding (MAE) tailored for 3D understanding. In this work, we continue on the path proposed by CroCo and focus on learning features tailored for 3D vision. In a nutshell, we extend MAE to arbitrarily many views of the same scene. By uniformly masking all views and employing a lightweight decoder with inter-frame attention, our approach is inherently simpler and more scalable than CroCo. We evaluate the resulting model, MuM, extensively on downstream tasks including feedforward reconstruction, dense image matching and relative pose estimation, finding that it outperforms the state-of-the-art visual encoders DINOv3 and CroCo v2.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。