arXiv:2502.09150cs.LGcs.CV2025-02被引 3

对比三类视觉模型对数据捷径的依赖程度,发现ViT最易被干扰。

Shortcut Learning Susceptibility in Vision Classifiers

  • 在数据中人为引入位置和强度捷径,测试模型是否依赖虚假线索
  • ViT在无位置编码时几乎完全忽略真实特征,准确率下降至45%
  • 低学习率下CNN更抗拒捷径,适合关注鲁棒性的研究者参考

快捷学习指模型利用数据中的虚假相关性而非真实特征,严重影响模型的泛化能力。本文系统评估了基于CNN、MLP和ViT的视觉分类器对快捷学习的敏感性。通过在数据集中引入与类别相关的显式位置和强度捷径,构建受控实验环境,考察模型是否依赖这些人工线索而非真实特征。采用定量评估:在含捷径的数据集上训练,分别在含捷径和不含捷径的测试集上测试;同时使用网络反演重建技术进行定性分析,观察模型权重中内化的信息。此外,还考察不同学习率下的敏感性。结果表明,低学习率下CNN更倾向于避免完全依赖捷径,而无位置编码的ViT几乎完全忽略真实图像特征,在捷径测试集上准确率降至45%。

原文摘要 · Abstract (English)

Shortcut learning, where machine learning models exploit spurious correlations in data instead of capturing meaningful features, poses a significant challenge to building robust and generalizable models. This phenomenon is prevalent across various machine learning applications, including vision, natural language processing, and speech recognition, where models may find unintended cues that minimize training loss but fail to capture the underlying structure of the data. Vision classifiers based on Convolutional Neural Networks (CNNs), Multi-Layer Perceptrons (MLPs), and Vision Transformers (ViTs) leverage distinct architectural principles to process spatial and structural information, making them differently susceptible to shortcut learning. In this study, we systematically evaluate these architectures by introducing deliberate shortcuts into the dataset that are correlated with class labels both positionally and via intensity, creating a controlled setup to assess whether models rely on these artificial cues or learn actual distinguishing features. We perform both quantitative evaluation by training on the shortcut-modified dataset and testing on two different test sets-one containing the same shortcuts and another without them-to determine the extent of reliance on shortcuts. Additionally, qualitative evaluation is performed using network inversion-based reconstruction techniques to analyze what the models internalize in their weights, aiming to reconstruct the training data as perceived by the classifiers. Further, we evaluate susceptibility to shortcut learning across different learning rates. Our analysis reveals that CNNs at lower learning rates tend to be more reserved against entirely picking up shortcut features, while ViTs, particularly those without positional encodings, almost entirely ignore the distinctive image features in the presence of shortcuts.

视觉分类快捷学习ViT鲁棒性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。