arXiv:2606.30444stat.MLcs.LG2026-06

SGD会优先学习伪特征,且速度呈指数级增长。

SGD Provably Prioritizes a Shortcut Spurious Feature in the XOR Model

论文配图:SGD Provably Prioritizes a Shortcut Spurious Feature in the XOR Model
图 1 · 摘自论文原文
  • 用在线小批量SGD训练两层ReLU网络,分析其学习过程
  • 伪特征学习速度指数级快,且抑制信号特征的提取
  • 适合研究模型偏见与优化机制的学者参考

神经网络容易过度依赖伪相关性,但其具体机制尚不明确,现有缓解方法多基于未经验证的假设。本文首次对两层ReLU网络在逻辑损失下使用在线小批量SGD训练时的伪特征学习行为提供了端到端理论分析。数据来自高维布尔超立方体,信号函数为二次型(即XOR),存在线性伪相关。结果表明:SGD首先以指数速度学习伪特征;优化动态使伪特征与信号特征耦合,强伪特征会抑制信号特征的学习。分析揭示了精确的相变现象:第一阶段,伪特征与第二层权重符号对齐导致其快速增长;第二阶段,多数类边界增大减缓学习,信号特征持续被压制。当伪相关最强时,即使在样本复杂度阈值处(孤立情况下可学得XOR),伪特征仍占主导。相反,若相关强度恒定,初步实证显示模型最终可学得XOR信号,但伪特征未被遗忘。

原文摘要 · Abstract (English)

Neural networks are known to be susceptible to over-reliance on spurious correlations. However, the precise mechanism by which models exploit shortcut features is not fully understood, and algorithms to mitigate this behavior rely on as yet unjustified assumptions about the learned representations. In this work, we provide the first end-to-end theoretical characterization of spurious feature learning for two-layer ReLU neural networks trained by online minibatch SGD on the logistic loss. We consider data drawn from the high-dimensional Boolean hypercube with a quadratic signal function (namely XOR) and a linear spurious correlation. We show that SGD learns the spurious feature first, and exponentially fast. Moreover, the optimization dynamics couple the spurious and signal features, with a stronger spurious component inhibiting signal feature learning. Our analysis reveals precise phase transitions in the learning dynamics. In the first phase, alignment between the signs of the spurious feature and second-layer weight drives rapid growth of the spurious feature. In the second phase, large majority group margin slows learning and the signal feature remains suppressed. When the spurious correlation is maximally strong, we show theoretically that the spurious feature dominates even at the sample complexity threshold where XOR would be learned in isolation (i.e., if the spurious feature was absent). In contrast, when the correlation strength is constant, we provide preliminary empirical evidence that the model can eventually learn the XOR signal, although the spurious feature is not forgotten.

深度学习优化机制伪特征

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。