用高斯点云解析视觉模型对3D世界的理解能力,无需3D标注数据。
Feat2GS: Probing Visual Foundation Models with Gaussian Splatting
- 从无姿态图像中提取特征生成3D高斯点云,实现对3D感知的统一评估。
- 在多个数据集上验证了视觉基础模型的几何与纹理感知能力,发现训练目标影响关键。
- 方法无需3D真值,可作为新视角合成的高效基线,适合研究3D认知的学者。
由于视觉基础模型(VFMs)在海量2D图像上训练,其对3D世界的理解能力尚不明确。现有3D探针方法依赖单视图2.5D估计或双视图稀疏2D对应,既忽略纹理感知,又需3D真值数据,限制了评估规模与多样性。为此,本文提出Feat2GS,通过读取未标定图像中VFM特征生成3D高斯点云(3DGS),从而在无需3D真值的前提下,利用新视角合成任务探测几何与纹理感知能力。3DGS参数的解耦(几何:$oldsymbol{x}$, $α$, $Σ$;纹理:$oldsymbol{c}$)支持对二者分别分析。在多种数据集上对多个VFMs的系统实验表明,训练目标是决定3D感知的关键因素。基于此,我们设计了若干改进变体,在多数据集上达到最优性能。代码与数据已开源。
原文摘要 · Abstract (English)
Given that visual foundation models (VFMs) are trained on extensive datasets but often limited to 2D images, a natural question arises: how well do they understand the 3D world? With the differences in architecture and training protocols (i.e., objectives, proxy tasks), a unified framework to fairly and comprehensively probe their 3D awareness is urgently needed. Existing works on 3D probing suggest single-view 2.5D estimation (e.g., depth and normal) or two-view sparse 2D correspondence (e.g., matching and tracking). Unfortunately, these tasks ignore texture awareness, and require 3D data as ground-truth, which limits the scale and diversity of their evaluation set. To address these issues, we introduce Feat2GS, which readout 3D Gaussians attributes from VFM features extracted from unposed images. This allows us to probe 3D awareness for geometry and texture via novel view synthesis, without requiring 3D data. Additionally, the disentanglement of 3DGS parameters - geometry ($\boldsymbol{x}$, $α$, $Σ$) and texture ($\boldsymbol{c}$) - enables separate analysis of texture and geometry awareness. Under Feat2GS, we conduct extensive experiments to probe the 3D awareness of several VFMs, and investigate the ingredients that lead to a 3D aware VFM. Building on these findings, we develop several variants that achieve state-of-the-art across diverse datasets. This makes Feat2GS useful for probing VFMs, and as a simple-yet-effective baseline for novel-view synthesis. Code and data are available at https://fanegg.github.io/Feat2GS/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。