只传注意力模式，模型也能学好特征，效果接近微调。
On the Surprising Effectiveness of Attention Transfer for Vision Transformers
- 用预训练ViT的注意力模式指导学生模型，不传特征
- 在ImageNet上达到与微调相当的准确率
- 适合想节省计算资源的研究者和部署场景
主流观点认为，预训练视觉Transformer（ViT）通过学习有用表征来提升下游性能。我们对此提出质疑并发现：预训练中学习到的特征与表征并非必需。令人惊讶的是，仅使用预训练模型的注意力模式（即信息在标记间流动的引导方式），学生模型即可从零开始学习高质量特征，并实现与微调相当的下游表现。为此，我们提出一种简单方法——注意力迁移：将预训练教师ViT的注意力图直接复制或蒸馏给学生。由于学生可自主学习特征，将其与微调后的教师模型集成后，进一步提升了ImageNet上的准确率。我们系统研究了注意力图充分性的多个方面，包括分布偏移设置下其性能低于微调的情况。本工作有助于理解预训练的实际作用，并提供了一种标准微调的可行替代方案。
原文摘要 · Abstract (English)
Conventional wisdom suggests that pre-training Vision Transformers (ViT) improves downstream performance by learning useful representations. Is this actually true? We investigate this question and find that the features and representations learned during pre-training are not essential. Surprisingly, using only the attention patterns from pre-training (i.e., guiding how information flows between tokens) is sufficient for models to learn high quality features from scratch and achieve comparable downstream performance. We show this by introducing a simple method called attention transfer, where only the attention patterns from a pre-trained teacher ViT are transferred to a student, either by copying or distilling the attention maps. Since attention transfer lets the student learn its own features, ensembling it with a fine-tuned teacher also further improves accuracy on ImageNet. We systematically study various aspects of our findings on the sufficiency of attention maps, including distribution shift settings where they underperform fine-tuning. We hope our exploration provides a better understanding of what pre-training accomplishes and leads to a useful alternative to the standard practice of fine-tuning
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。