用中间层特征提升模型对分布外数据的泛化能力
Intermediate Layer Classifiers for OOD generalization
- 提出中间层分类器，利用网络早期层特征进行分布外预测
- 早期层特征在多种分布偏移下表现优于最后一层
- 无需微调即可实现接近微调后的零样本泛化性能
深度分类器易受数据分布变化影响，主要因其依赖训练数据中的虚假相关。尽管有人认为最后一层仍能提取稳定特征，但本文质疑其有效性，转而探索中间层的作用。我们提出中间层分类器（ILCs），发现中间层表示在多数情况下显著优于倒数第二层。在多种数据集、架构和分布偏移下，使用早期层特征的零样本泛化性能接近于在倒数第二层微调后的少样本表现。分析表明，中间层对分布偏移的敏感性低于倒数第二层。结果强调了理解网络各层信息分布对分布外泛化的重要性，也揭示了倒数第二层表示的局限性。代码已开源。
原文摘要 · Abstract (English)
Deep classifiers are known to be sensitive to data distribution shifts, primarily due to their reliance on spurious correlations in training data. It has been suggested that these classifiers can still find useful features in the network's last layer that hold up under such shifts. In this work, we question the use of last-layer representations for out-of-distribution (OOD) generalisation and explore the utility of intermediate layers. To this end, we introduce \textit{Intermediate Layer Classifiers} (ILCs). We discover that intermediate layer representations frequently offer substantially better generalisation than those from the penultimate layer. In many cases, zero-shot OOD generalisation using earlier-layer representations approaches the few-shot performance of retraining on penultimate layer representations. This is confirmed across multiple datasets, architectures, and types of distribution shifts. Our analysis suggests that intermediate layers are less sensitive to distribution shifts compared to the penultimate layer. These findings highlight the importance of understanding how information is distributed across network layers and its role in OOD generalisation, while also pointing to the limits of penultimate layer representation utility. Code is available at https://github.com/oshapio/intermediate-layer-generalization
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。