arXiv:2608.18399cs.CV2026-08

注意力迁移不能复制模型的鲁棒性,真正缺失的是特征知识。

What Does Attention Transfer Transfer? Attention Structure and Robustness in Vision Transformers

论文配图:What Does Attention Transfer Transfer? Attention Structure and Robustness in Vision Transformers
图 1 · 摘自论文原文
  • 通过结构化追踪发现,学生模型注意力与教师几乎完全一致
  • 在数据量和参数量减少10倍时,鲁棒性仍落后于教师模型
  • 直接干预注意力结构也无法提升鲁棒性,说明问题不在注意力本身

视觉变换器(ViT)通过复制预训练教师的注意力图,在分布内任务上恢复了大部分微调精度,但在分布外情况下仍明显逊色。本文针对基于ImageNet-100的自监督教师与ViT-S学生模型,构建了注意力结构的测量工具,得出三个结论:第一,注意力转移几乎完美且稳定,学生注意力与教师距离比微调结果小两个数量级,且不随训练继续而变化;第二,在参数少14倍、数据少10倍的情况下,差距存在但具有时间维度——鲁棒性成熟晚于准确率,按准确率停止规则会低估鲁棒性;第三,将注意力结构冗余减半后,即便在准确率匹配条件下,也未观察到鲁棒性变化。三者共同表明,性能差距源于特征而非可见注意力结构。在此测量范围内,注意力图只反映‘看’的位置,不反映‘知道’的内容。

原文摘要 · Abstract (English)

Vision transformers (ViTs) trained to copy a pretrained teacher's attention maps recover most of fine-tuning's in-distribution accuracy yet fall measurably short of it under distribution shift, as recent work has shown. What the copy delivers has never been measured directly in the attention structure and tied to robustness. We build that instrumentation for ViT-S students of a self-supervised teacher on ImageNet-100, and report three findings that triangulate one conclusion. First, the transfer is essentially perfect and permanently so: the distilled student's attention ends up roughly two orders of magnitude closer to the teacher's than fine-tuning does, and does not drift with additional training. Second, the gap is real at 14$\times$ fewer parameters and 10$\times$ less data than previously studied, but it has a time axis. It tracks training maturity, and completing the schedules that the stopping rule interrupted closes it below our pre-registered threshold in two of three seeds, with comparisons at equal accuracy giving the same result. The endpoint gap at this scale is substantially a training-maturity artifact: robustness matures later than accuracy, and stopping rules tuned to accuracy undersample it. Third, forcing cross-row redundancy down by half the structural separation between the distilled and fine-tuned conditions produces no detectable robustness response under two registered ways of matching accuracy. Verified transfer, a gap that closes while the structure never moves, and a null under direct intervention are together consistent with the deficit residing in features, not in the visible attention structure. This is elimination plus intervention, and its scope is the regime we measured. In this regime, attention overlays show where a model looks, not what it knows.

视觉变换器注意力机制模型鲁棒性知识蒸馏

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。