发现ViT依赖背景补丁偷懒，提出改进方法提升12个任务表现
Vision Transformers Need More Than Registers
- 通过分析发现ViT用无关背景块代替全局语义
- 新方法减少背景干扰，12个基准测试性能提升
- 适合研究ViT机制或想提升模型泛化性的读者
视觉变换器（ViTs）在大规模数据预训练后，可为多种下游任务提供通用表征。然而，不同监督范式和下游任务中广泛存在模型缺陷。本文通过系统分析发现，这些缺陷源于一种‘偷懒聚合’行为：由于全局注意力和粗粒度语义监督，ViT会利用语义无关的背景补丁作为捷径来表征全局信息。为此，我们提出选择性地将补丁特征融合至CLS token，削弱背景主导的捷径影响，在标签监督、文本监督和自监督三种设置下，于12个基准测试中均实现性能提升。本工作为理解ViT行为提供了新视角。
原文摘要 · Abstract (English)
Vision Transformers (ViTs), when pre-trained on large-scale data, provide general-purpose representations for diverse downstream tasks. However, artifacts in ViTs are widely observed across different supervision paradigms and downstream tasks. Through systematic analysis of artifacts in ViTs, we find that their fundamental mechanisms have yet to be sufficiently elucidated. In this paper, through systematic analysis, we conclude that these artifacts originate from a lazy aggregation behavior: ViT uses semantically irrelevant background patches as shortcuts to represent global semantics, driven by global attention and Coarse-grained semantic supervision. Our solution selectively integrates patch features into the CLS token, reducing the influence of background-dominated shortcuts and consistently improving performance across 12 benchmarks under label-, text-, and self-supervision. We hope this work offers a new perspective on ViT behavior.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。