发现ImageNet模型依赖高频纹理特征,影响跨域泛化能力
Do ImageNet-trained models learn shortcuts? The impact of frequency shortcuts on generalization
- 提出高效方法大规模分析模型对频率特征的依赖
- 发现CNN与Transformer在ImageNet上均学习高频纹理捷径
- 提醒评估基准应考虑频率捷径对泛化的影响
频率捷径指模型在分类时过度依赖特定频率模式。已有研究表明,小规模数据集训练的模型常利用此类捷径,损害泛化性能。然而,现有识别方法计算成本高,难以应用于大规模数据训练的模型。本文首次提出可高效扩展的频率捷径分析方法,发现无论CNN还是Transformer,在ImageNet上均存在频率捷径。这些捷径主要对应纹理模式,在保留纹理信息的分布外(OOD)测试集上表现良好,却显著阻碍模型在重绘类OOD测试集上的泛化能力。这表明当前的OOD评估常忽视频率捷径的影响。未来基准应显式评估并纳入此类因素,以构建更鲁棒的泛化模型。
原文摘要 · Abstract (English)
Frequency shortcuts refer to specific frequency patterns that models heavily rely on for correct classification. Previous studies have shown that models trained on small image datasets often exploit such shortcuts, potentially impairing their generalization performance. However, existing methods for identifying frequency shortcuts require expensive computations and become impractical for analyzing models trained on large datasets. In this work, we propose the first approach to more efficiently analyze frequency shortcuts at a large scale. We show that both CNN and transformer models learn frequency shortcuts on ImageNet. We also expose that frequency shortcut solutions can yield good performance on out-of-distribution (OOD) test sets which largely retain texture information. However, these shortcuts, mostly aligned with texture patterns, hinder model generalization on rendition-based OOD test sets. These observations suggest that current OOD evaluations often overlook the impact of frequency shortcuts on model generalization. Future benchmarks could thus benefit from explicitly assessing and accounting for these shortcuts to build models that generalize across a broader range of OOD scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。