揭示了神经网络偏好捷径特征的数学机制，发现其与核函数特征值有关。
Shortcut Features as Top Eigenfunctions of NTK: A Linear Neural Network Case and More
- 用NTK框架将特征定义为核函数的特征函数，分析线性网络中的捷径学习。
- 样本不均衡时，高特征值对应捷径特征，且训练后仍主导输出结果。
- 该现象在多层ReLU网络和ResNet-18中得到验证，说明非最大间隔是主因。
深度学习模型常出现捷径学习问题：当多数训练数据被某一特定特征主导时，神经网络会优先学习该特征，即使它不具备泛化能力。本文基于神经正切核（NTK）框架，研究线性神经网络，将网络特征定义为NTK的特征函数。发现当捷径源于聚类分布中的样本数量不均衡时，捷径特征对应于具有更大特征值的特征函数。此外，由于簇内数据方差的存在，这些高特征值特征在训练后仍对网络输出有显著影响。即使控制输出的间隔，这种对特定特征的偏好依然存在，表明最大间隔偏差并非导致捷径学习的唯一主要原因。上述线性网络中的性质，通过两层全连接ReLU网络和ResNet-18在实验上得到扩展验证。
原文摘要 · Abstract (English)
One of the chronic problems of deep-learning models is shortcut learning. In a case where the majority of training data are dominated by a certain feature, neural networks prefer to learn such a feature even if the feature is not generalizable outside the training set. Based on the framework of Neural Tangent Kernel (NTK), we analyzed the case of linear neural networks to derive some important properties of shortcut learning. We defined a feature of a neural network as an eigenfunction of NTK. Then, we found that shortcut features correspond to features with larger eigenvalues when the shortcuts stem from the imbalanced number of samples in the clustered distribution. We also showed that the features with larger eigenvalues still have a large influence on the neural network output even after training, due to data variances in the clusters. Such a preference for certain features remains even when a margin of a neural network output is controlled, which shows that the max-margin bias is not the only major reason for shortcut learning. These properties of linear neural networks are empirically extended for more complex neural networks as a two-layer fully-connected ReLU network and a ResNet-18.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。