发现视觉大模型对视角变化敏感，易因几何相似误判物体。
Not all Views are Created Equal: Analyzing Viewpoint Instabilities in Vision Foundation Models
- 仅用特征表示识别视角是否异常或稳定，无需原始图像。
- 九个模型中，部分在视角微变时特征剧烈波动，导致3D推理偏差。
- 适合关注3D理解鲁棒性、视觉模型泛化能力的研究者。
本文分析视觉基础模型在视角变化下的稳定性，定义视角不稳定性为微小视角变动引发显著特征变化，进而造成3D推理任务中的泛化差距。研究涵盖九个基础模型，重点关注其对视角变化的响应，包括常被忽视的意外视角（特定相机朝向会遮蔽物体真实3D结构）。方法仅通过特征表示即可识别并分类分布外（OOD）、意外及稳定视角，无需访问实际图像。结果表明，尽管所有模型均编码意外视角，但对分布外视角的解释存在差异，受固有偏见影响，有时依据几何相似性错误分类物体。在分类、VQA和3D重建三个下游任务上，定量与定性评估揭示了视角不稳定性的影响，并强调了特征在多样视角条件下的鲁棒性重要性。
原文摘要 · Abstract (English)
In this paper, we analyze the viewpoint stability of foundational models - specifically, their sensitivity to changes in viewpoint- and define instability as significant feature variations resulting from minor changes in viewing angle, leading to generalization gaps in 3D reasoning tasks. We investigate nine foundational models, focusing on their responses to viewpoint changes, including the often-overlooked accidental viewpoints where specific camera orientations obscure an object's true 3D structure. Our methodology enables recognizing and classifying out-of-distribution (OOD), accidental, and stable viewpoints using feature representations alone, without accessing the actual images. Our findings indicate that while foundation models consistently encode accidental viewpoints, they vary in their interpretation of OOD viewpoints due to inherent biases, at times leading to object misclassifications based on geometric resemblance. Through quantitative and qualitative evaluations on three downstream tasks - classification, VQA, and 3D reconstruction - we illustrate the impact of viewpoint instability and underscore the importance of feature robustness across diverse viewing conditions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。