提出WWT模型,让视觉模型同时看清物体是什么和在哪里。
What-Where Transformer: A Slot-Centric Visual Backbone for Concurrent Representation and Localization

- 用多路槽结构并行处理物体特征和位置信息
- 在ImageNet上实现无需后处理的多物体发现
- 适合需要精准定位的任务,如分割与检测
许多图像理解任务需要识别物体的类别(what)及其位置(where)。然而,定位类任务通常比仅识别类别的分类任务更复杂,原因在于传统分类骨干网络倾向于强化语义信息而弱化空间位置信息。本文提出一种“所见-所在分离”的归纳偏置,鼓励模型以解耦方式表示物体外观与空间位置。为此,我们设计了What-Where Transformer(WWT),其核心创新为:(1)将注意力图视为位置表示,令牌视为外观表示,并通过多流槽式架构在前馈模块中并行处理;(2)复用最终层的令牌与注意力图参与下游任务,并直接暴露于任务损失梯度,促进显式定位学习。实验表明,在标准ImageNet单标签分类监督下,WWT能从原始注意力图中直接涌现出多物体发现能力,无需额外聚类等后处理。此外,相比基于ViT的方法,WWT在零样本物体发现和弱监督语义分割任务上表现更优,且可轻松迁移至多种定位场景,仅需少量修改。代码将在录用后公开。
原文摘要 · Abstract (English)
Many image understanding tasks involve identifying what is present and where it appears. However, tasks that address where, such as object discovery, detection, and segmentation, are often considerably more complex than image classification, which primarily focuses on what. One possible reason is that classification-oriented backbones tend to emphasize semantic information about what, while implicitly entangling or suppressing information about where. In this work, we focus on an inductive bias termed what-where separation, which encourages models to represent object appearance and spatial location in a decomposed manner. To incorporate this bias throughout an attentive backbone in the style of Vision Transformer (ViT), we propose the What-Where Transformer (WWT). Our method introduces two key novel designs: (1) it treats tokens as representations of what and attention maps as representations of where, and processes them in concurrent feed-forward modules via a multi-stream, slot-based architecture; (2) it reuses both the final-layer tokens and attention maps for downstream tasks, and directly exposes them to gradients derived from task losses, thereby facilitating more effective and explicit learning of localization. We demonstrate that even under standard single-label classification-based supervision on ImageNet, WWT exhibits emergent multiple object discovery directly from raw attention maps, rather than via additional postprocessing such as token clustering. Furthermore, WWT achieves superior performance compared to ViT-based methods on zero-shot object discovery and weakly supervised semantic segmentation, and it is transferable to various localization setups with minimal modifications. Code will be published after acceptance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。