用2D视觉模型提取3D形状特征，无需3D标注也能达到顶尖效果
MeshFM: 2D Features Are All You Need for 3D Shape Understanding
- 从2D视觉模型蒸馏特征到3D，用前馈网络直接预测3D特征
- 仅用2D监督训练，在零件分割等任务上媲美3D监督方法
- 对极端旋转鲁棒，适合无3D标注的3D理解场景
我们提出MeshFM，一种高效的前馈框架，用于从3D输入中提取丰富特征。该方法将视觉基础模型中的2D特征提炼至3D空间。通过两阶段训练策略：首先仅使用2D特征监督优化3D特征场；其次训练网络回归该特征场。整个过程无需3D标注，完全依赖2D基础模型的强大信息。实验表明，所学特征可直接应用于零件分割、密集对应和网格变形等下游任务。大量实验显示，仅通过2D监督训练的MeshFM，在无需任务微调的情况下，性能与使用显式3D监督的方法相当。此外，模型对输入物体的极端旋转具有鲁棒性。
原文摘要 · Abstract (English)
We present MeshFM, an efficient feedforward framework for extracting rich features from 3D inputs. Our method distills 2D features from visual foundation models into 3D. We train a feedforward network to directly predict 3D features without requiring optimization during inference. The approach utilizes a two-stage training strategy. First, we optimize a feature field in 3D using only 2D feature supervision. Second, we train a network to regress this feature field. The entire procedure requires no 3D annotation, instead relying on the powerful information in 2D foundation models. We demonstrate that our learned features can be immediately applied to downstream tasks, including part segmentation, dense correspondence, and mesh deformation. Extensive experiments show that MeshFM, trained solely with 2D supervision, performs on par with methods trained explicitly with 3D supervision, even without task-specific fine-tuning. Moreover, our model is trained to be robust to extreme rotations of the input objects. Project page: https://threedle.github.io/MeshFM/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。