arXiv:2603.26127cs.CVcs.AI2026-03被引 1

发现并利用ViT中分散的物体中心信息,提升无监督物体发现效果。

Finding Distributed Object-Centric Properties in Self-Supervised Transformers

  • 通过分析所有层的查询、键、值注意力相似性,挖掘物体中心特征。
  • 在无监督物体发现任务上提升3.6至12.4个点的定位准确率。
  • 无需额外训练,适用于物体识别与多模态模型去幻觉场景。

自监督视觉变压器(如DINO)虽能涌现发现物体的能力,但其最终层[CLS]令牌的注意力图常存在虚假激活,导致定位不准。这是因为[CLS]令牌基于图像级目标训练,汇总整图信息,稀释了局部块间交互中的物体中心信息。我们通过计算各层块间注意力组件(查询、键、值)的相似性,发现:(1) 物体中心特性编码于三者生成的相似性图中,而不仅限于键特征或[CLS]令牌;(2) 这类信息分布于网络全层,并非仅存于末层。基于此,我们提出Object-DINO——一种无需训练的方法,通过聚类跨层注意力头,自动识别对应所有物体的物体中心簇。我们在两项任务中验证其有效性:在无监督物体发现中实现+3.6至+12.4的CorLoc提升,以及通过提供视觉锚定,缓解多模态大模型中的物体幻觉问题。结果表明,利用分布式物体中心信息可有效提升下游任务表现,且无需额外训练。

原文摘要 · Abstract (English)

Self-supervised Vision Transformers (ViTs) like DINO show an emergent ability to discover objects, typically observed in [CLS] token attention maps of the final layer. However, these maps often contain spurious activations resulting in poor localization of objects. This is because the [CLS] token, trained on an image-level objective, summarizes the entire image instead of focusing on objects. This aggregation dilutes the object-centric information existing in the local, patch-level interactions. We analyze this by computing inter-patch similarity using patch-level attention components (query, key, and value) across all layers. We find that: (1) Object-centric properties are encoded in the similarity maps derived from all three components ($q, k, v$), unlike prior work that uses only key features or the [CLS] token. (2) This object-centric information is distributed across the network, not just confined to the final layer. Based on these insights, we introduce Object-DINO, a training-free method that extracts this distributed object-centric information. Object-DINO clusters attention heads across all layers based on the similarities of their patches and automatically identifies the object-centric cluster corresponding to all objects. We demonstrate Object-DINO's effectiveness on two applications: enhancing unsupervised object discovery (+3.6 to +12.4 CorLoc gains) and mitigating object hallucination in Multimodal Large Language Models by providing visual grounding. Our results demonstrate that using this distributed object-centric information improves downstream tasks without additional training.

自监督物体发现注意力机制ViT

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。