发现视觉Transformer中间层性能受数据分布差异影响，按模块选择探测方式可提升泛化能力。
Layer by layer, module by module: Choose both for optimal OOD probing of ViT
- 分层分模块探测：根据分布偏移程度选择不同模块激活值进行评估
- 深层特征在分布偏移时性能下降，浅层更鲁棒；弱偏移时注意力输出最优
- 突破传统统一探测方法，适合关注模型泛化性的研究者
近期研究发现，基础模型的中间层常比最终层具有更强判别性表征。尽管最初归因于自回归预训练，这一现象也在监督和判别式自监督训练模型中被观察到。本文对预训练视觉变换器的中间层行为进行了全面分析。通过在多样图像分类基准上开展大量线性探测实验，我们发现预训练与下游数据之间的分布偏移是深层性能退化的主要原因。进一步在模块层面进行细粒度分析发现，标准的变压器块输出探测效果不佳；当存在显著分布偏移时，探测前馈网络内的激活值表现最佳；而分布偏移较弱时，多头自注意力模块的归一化输出最优。
原文摘要 · Abstract (English)
Recent studies have observed that intermediate layers of foundation models often yield more discriminative representations than the final layer. While initially attributed to autoregressive pretraining, this phenomenon has also been identified in models trained via supervised and discriminative self-supervised objectives. In this paper, we conduct a comprehensive study to analyze the behavior of intermediate layers in pretrained vision transformers. Through extensive linear probing experiments across a diverse set of image classification benchmarks, we find that distribution shift between pretraining and downstream data is the primary cause of performance degradation in deeper layers. Furthermore, we perform a fine-grained analysis at the module level. Our findings reveal that standard probing of transformer block outputs is suboptimal; instead, probing the activation within the feedforward network yields the best performance under significant distribution shift, whereas the normalized output of the multi-head self-attention module is optimal when the shift is weak.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。