自监督视觉模型能像人一样分组物体,关键在特征相似性结构。
Human-like Object Grouping in Self-supervised Vision Transformers
- 用人类判断物体相同/不同的实验,测试模型对物体的感知能力。
- 基于DINO训练的Transformer模型最接近人类反应,且特征内聚性越强越准。
- 通过匹配特征相似性矩阵(Gram矩阵),可提升模型与人类感知的一致性。
以自监督目标训练的视觉基础模型在多种任务中表现优异,并展现出涌现的物体分割特性。然而,其与人类物体感知的对齐程度仍不清晰。本文引入一个行为基准,让参与者在自然场景中对点对进行相同/不同判断,将经典心理物理学范式扩展至超过1000次试验。我们测试了多种视觉模型,通过简单读出其表征来预测人类反应时间。观察到模型代际间持续改进,架构与训练目标均贡献于对齐,其中基于Transformer且采用DINO自监督目标的模型表现最佳。为探究改进来源,我们提出一种新度量方法,通过测量图像块内与块间相似性来量化表征中的对象中心成分。结果显示,对象中心结构越强,越能准确预测人类分割行为。进一步表明,通过蒸馏使自监督模型的Gram矩阵与监督式Transformer模型对齐,可提升其与人类行为的契合度,这与先前发现一致:Gram锚定能改善DINOv3的特征质量。综合结果表明,自监督视觉模型以行为上类人的形式捕捉物体结构,且Gram矩阵结构在推动感知对齐中起作用。
原文摘要 · Abstract (English)
Vision foundation models trained with self-supervised objectives achieve strong performance across diverse tasks and exhibit emergent object segmentation properties. However, their alignment with human object perception remains poorly understood. Here, we introduce a behavioral benchmark in which participants make same/different object judgments for dot pairs on naturalistic scenes, scaling up a classical psychophysics paradigm to over 1000 trials. We test a diverse set of vision models using a simple readout from their representations to predict subjects' reaction times. We observe a steady improvement across model generations, with both architecture and training objective contributing to alignment, and transformer-based models trained with the DINO self-supervised objective showing the strongest performance. To investigate the source of this improvement, we propose a novel metric to quantify the object-centric component of representations by measuring patch similarity within and between objects. Across models, stronger object-centric structure predicts human segmentation behavior more accurately. We further show that matching the Gram matrix of supervised transformer models, capturing similarity structure across image patches, with that of a self-supervised model through distillation improves their alignment with human behavior, converging with the prior finding that Gram anchoring improves DINOv3's feature quality. Together, these results demonstrate that self-supervised vision models capture object structure in a behaviorally human-like manner, and that Gram matrix structure plays a role in driving perceptual alignment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。