通过多视图掩码重建，让模型学会3D几何理解，无需3D标注。
Muskie: Multi-view Masked Image Modeling for 3D Vision Pre-training
- 多视图并行处理，用其他视图信息恢复被遮蔽内容。
- 在相机位姿估计和点云重建任务上超越现有模型。
- 适合做3D视觉预训练，尤其无监督场景下表现优异。
我们提出Muskie，一种专为3D视觉任务设计的原生多视图视觉骨干网络。与现有逐帧处理模型不同，Muskie能同时处理多个视角，并在预训练阶段引入多视图一致性。该模型通过从其他视图中寻找几何对应关系，重建某一视图中被严重遮蔽的内容。借助这一预训练任务和提出的激进掩码策略，模型隐式学习到视图不变特征并建立强几何理解，全程无需任何3D监督信号。相比最先进的逐帧骨干网络如DINO，Muskie在多视图对应精度上表现更优。此外，将Muskie作为骨干应用于下游3D任务，包括相机位姿估计和点云重建，均取得一致性能提升。代码已公开于 https://leo-frank.github.io/Muskie/
原文摘要 · Abstract (English)
We present Muskie, a native multi-view vision backbone designed for 3D vision tasks. Unlike existing models, which are frame-wise and exhibit limited multi-view consistency, Muskie is designed to process multiple views simultaneously and introduce multi-view consistency in pre-training stage. Muskie is trained to reconstruct heavily masked content in one view by finding and utilizing geometric correspondences from other views. Through this pretext task and our proposed aggressive masking strategy, the model implicitly to learn view-invariant features and develop strong geometric understanding without any 3D supervision. Compared with state-of-the-art frame-wise backbones such as DINO, Muskie achieves higher multi-view correspondence accuracy. Furthermore, we demonstrate that using Muskie as a backbone consistently enhances performance on downstream 3D tasks, including camera pose estimation and pointmap reconstruction. Codes are publicly available at https://leo-frank.github.io/Muskie/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。