数据不平衡反而提升模型泛化能力，关键在模型够强。
When Data Imbalance Helps: Robust Generalization Through Shortcut Saturation
- 用数据不平衡诱导模型避开捷径特征，转向真正规律。
- 当捷径特征与真实标签相关度达90%时，模型在对抗测试中准确率达100%。
- 适合研究模型如何摆脱虚假相关性的研究人员参考。
我们研究了在虚假相关性下的鲁棒泛化问题：训练集中某捷径特征与真实标签正相关，但在对抗性测试集上反相关。通过调节捷径比例 $r$（训练集中捷径特征等于真实标签的比例）和模型容量，发现一个反直觉现象：在足够强大的模型中，数据不平衡反而促进泛化。在一个合成任务中，真实标签是整数序列的奇偶和，捷径特征是最大值元素的奇偶性。当 $r=0.50$ 时，两层双头Transformer在0%的随机种子下达到100%对抗准确率；而当 $r=0.90$ 时，成功率升至77%。该现象在单层模型中不存在，数据不平衡反而使模型陷入捷径。通过梯度冲突动力学、电路演化及QK/OV电路消融分析，揭示了一条与不平衡促进泛化一致的机制路径。
原文摘要 · Abstract (English)
We study robust generalization under spurious correlations: tasks where a shortcut feature is correlated with the true label in training but anti-correlated in an adversarial held-out split. Varying the spurious ratio $r$ (the fraction of training examples where shortcut = true label) and model capacity, we find a counterintuitive result: data imbalance promotes generalization in sufficiently capable models. On a synthetic task where the true label is sum parity of an integer sequence and the shortcut is the parity of the maximum-valued element, a 2-layer, 2-head transformer generalized (reached $100\%$ adversarial accuracy) in 0% of seeds at $r{=}0.50$ but 77% of seeds at $r{=}0.90$. The effect is absent in 1-layer models, where imbalance instead traps the model on the shortcut. Through mechanistic analysis -- gradient conflict dynamics, circuit evolution, and QK/OV circuit ablations -- we characterize a mechanistic pathway consistent with imbalance promoting generalization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。