通过CutMix训练协议让ViT早期层产生局部注意力，无需预训练。
Inducing Spatial Locality in Vision Transformers through the Training Protocol
- 用CutMix等现代数据增强策略诱导视觉注意力集中于局部区域。
- 在CIFAR-100上，最小注意力距离从0.316降至0.008，显著提升局部性。
- 适用于希望不依赖预训练、提升ViT局部感知能力的研究者。
我们研究了训练协议是否能在无大规模预训练的情况下，使从零开始训练的视觉变换器（ViT）早期层产生空间局部性。在保持架构与优化过程不变的前提下，我们在CIFAR-10、CIFAR-100和Tiny-ImageNet上对比了基础协议与现代协议（AutoAugment/ColorJitter、CutMix和标签平滑），通过平均注意力距离（MAD）和归一化熵刻画每个注意力头。在三个数据集上，现代协议均使早期层注意力更局部且更集中；在CIFAR-100上，最小MAD由基础协议的0.316降至现代协议的0.008。进一步在CIFAR-100上进行消融实验，发现仅CutMix具有决定性作用：含CutMix的条件中MAD稳定在0.024，不含则维持在0.210；AutoAugment与标签平滑无独立影响。结果表明，由CutMix引发的对部分图像区域分类的压力，可促进视觉变换器中局部注意力的出现。
原文摘要 · Abstract (English)
We investigate whether the training protocol can induce spatial locality in the early layers of a Vision Transformer (ViT) trained from scratch, without large-scale pretraining. Keeping the architecture and optimization procedure fixed, we compare a Baseline protocol with a Modern protocol (AutoAugment/ColorJitter, CutMix, and Label Smoothing) on CIFAR-10, CIFAR-100, and Tiny-ImageNet, characterizing each attention head via Mean Attention Distance (MAD) and normalized entropy. Across all three datasets, the Modern protocol produces more local and more concentrated attention in early layers; on CIFAR-100, the minimum MAD drops from 0.316 (Baseline) to 0.008 (Modern). To identify the source of this effect, we conduct an ablation study on CIFAR-100 by adding or removing each component individually. The results identify CutMix as the determining component within our experiments: all conditions with CutMix exhibit MAD 0.024, while all conditions without CutMix remain at MAD 0.210. AutoAugment and Label Smoothing show no independent effect on locality. Taken together, these findings suggest that the pressure to classify from partial image regions, induced by CutMix, can promote the emergence of local attention in Vision Transformers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。