构建首个系统性物体对应关系评估基准,衡量视觉模型对物体部件的精细理解能力。
SOCO: Benchmarking Semantic Object Correspondence in Vision Foundation Models

- 提出SOCO基准,统一标注100类物体超过100万对部件对应关系。
- 发现主流视觉模型跨类别传递对应关系能力弱,且对部件位置建模不完整。
- 适用于评估大模型对细粒度部件的理解,尤其适合研究多模态与结构化表示的学者。
衡量视觉基础模型的结构化物体理解能力仍面临评估协议不一致和局部监督不足的挑战。语义对应(SC)通过测试在外观、视角和几何形变下跨实例与类别匹配物体部件的能力来评估该能力。为实现系统的SC评估,我们引入SOCO,一个全新的语义物体对应基准,包含对应类型分类体系,并在100个类别上提供超过100万对具有一致功能意义的关键点标注。此外,SOCO包含关键点的语言描述,支持大视觉语言模型(LVLMs)及其细粒度部件理解的评估。全面实验表明:(i) 视觉基础主干网络编码了强语义结构,但跨相关类别传递对应关系能力差,且仅部分捕捉部件位置;(ii) LVLMs在文本提示下的部件定位优于基于视觉参考的跨图像匹配,暴露出语言引导定位与细粒度视觉对应之间的差距;(iii) 对应性能比ImageNet分类更能预测分割、跟踪、3D姿态估计和3D检测等密集下游任务的表现。这些发现使SOCO成为视觉与多模态基础模型中结构化、部件级表征质量的基准。
原文摘要 · Abstract (English)
Measuring structured object understanding in vision foundation models remains challenging due to inconsistent evaluation protocols and limited part-level supervision. Semantic correspondence (SC) evaluates this capability by testing whether object parts can be matched across instances and categories under large variations in appearance, viewpoint, and geometry. To enable a systematic SC evaluation, we introduce SOCO, a new benchmark for Semantic Object Correspondence that introduces a taxonomy of correspondence types and provides consistent, functionally meaningful keypoint annotations across 100 categories and over 1M correspondence pairs. In addition, SOCO includes keypoint language descriptions, enabling the evaluation of large vision-language models (LVLMs) and their fine-grained part-level understanding. Comprehensive experiments reveal that (i) vision foundation backbones encode strong semantic structure but transfer correspondences poorly across related categories and only partially capture object-part position, (ii) LVLMs are stronger at text-prompted part localization than at visual-reference cross-image matching, exposing a gap between language-grounded localization and fine-grained visual correspondence, and (iii) correspondence performance predicts performance on dense downstream tasks, including segmentation, tracking, 3D pose estimation, and 3D detection, more strongly than ImageNet classification. Together, these findings position SOCO as a benchmark for structured, part-level representation quality in vision and multimodal foundation models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。