让视觉模型同时看多张图，让相同物体点的特征更一致。
Multi-View Foundation Models
- 在基础模型中加入3D感知注意力层，实现跨视角特征对齐。
- 在表面法线估计和多视图分割任务上显著提升匹配精度。
- 无需构建3D模型，直接在图像空间操作，适合多视角应用。
基础模型在计算机视觉中至关重要，通常以单张RGB图像为输入，输出可用于多种任务的深度特征表示。然而，当存在同一3D场景的多视角图像时，现有模型独立处理每张图像，常导致相同3D点的特征不一致。本文提出将基础模型转化为多视图基础模型的方法，使其接收一组图像输入，并输出每个图像对应的特征图，使对应点的特征尽可能一致。该方法避免了构建统一的3D特征模型，支持在图像空间直接操作。具体地，我们通过在基于Transformer的基础模型（如DINO、SAM、CLIP）中引入中间3D感知注意力层，增强跨视角特征匹配能力。以表面法线估计和多视图分割为例，定量实验表明，本方法在特征匹配方面显著优于现有基础模型。
原文摘要 · Abstract (English)
Foundation models are vital tools in various Computer Vision applications. They take as input a single RGB image and output a deep feature representation that is useful for various applications. However, in case we have multiple views of the same 3D scene, they operate on each image independently and do not always produce consistent features for the same 3D point. We propose a way to convert a Foundation Model into a Multi-View Foundation Model. Such a model takes as input a set of images and outputs a feature map for each image such that the features of corresponding points are as consistent as possible. This approach bypasses the need to build a consistent 3D model of the features and allows direct manipulation in the image space. Specifically, we show how to augment Transformers-based foundation models (i.e., DINO, SAM, CLIP) with intermediate 3D-aware attention layers that help match features across different views. As leading examples, we show surface normal estimation and multi-view segmentation tasks. Quantitative experiments show that our method improves feature matching considerably compared to current foundation models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。