不微调直接测试大模型3D理解能力，评估多视角图像对应关系。
Evaluating Foundation Models' 3D Understanding Through Multi-View Correspondence Analysis
- 基于多视角图像构建无微调的3D场景理解评测框架。
- 在MVImgNet数据集上按视图差异分四类，最高准确率达72.3%。
- 发现DINO系编码器在大视角变化下仍表现稳健，适合视觉推理任务。
评估基础模型的3D空间理解能力对机器人与自动驾驶等实际应用至关重要。现有评估常依赖下游微调或任务特定解码器，难以分离预训练编码器的内在3D推理能力。本文提出一种无需微调的上下文3D场景理解新基准，直接探测密集视觉特征质量。基于已有的Hummingbird框架（用于2D场景理解），我们将其扩展至3D多视角ImageNet（MVImgNet）数据集。给定一组特定视角拍摄的图像（键），评估模型对新视角（查询）的分割性能，并根据键-查询视角差异分为易、中、难、极难四类。我们评测了7个顶尖基础模型，结果显示基于DINO的编码器在大视角偏移下依然保持竞争力。代码已开源：https://github.com/ToyeshC/open-hummingbird-3d-eval。
原文摘要 · Abstract (English)
Benchmarking 3D spatial understanding of foundation models is essential for real-world applications such as robotics and autonomous driving. Existing evaluations often rely on downstream fine-tuning with linear heads or task-specific decoders, making it difficult to isolate the intrinsic 3D reasoning ability of pre-trained encoders. In this work, we introduce a novel benchmark for in-context 3D scene understanding that requires no fine-tuning and directly probes the quality of dense visual features. Building on the Hummingbird framework, which evaluates in-context 2D scene understanding, we extend the setup to the 3D Multi-View ImageNet (MVImgNet) dataset. Given a set of images depicting objects at specific camera angles (keys), we benchmark the performance of segmenting novel views (queries) and report the scores in 4 categories of easy, medium, hard, and extreme based on the key-query view contrast. We benchmark 7 state-of-the-art foundation models and show that DINO-based encoders remain competitive across large viewpoint shifts. Our code is publicly available at https://github.com/ToyeshC/open-hummingbird-3d-eval.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。