研究注意力模型过拟合为何无害，发现两步梯度下降即可实现最优泛化。
Benign Overfitting in Single-Head Attention
- 在单头注意力模型中，用两步梯度下降即可实现完美拟合且泛化良好。
- 信号噪声比足够高时，模型能同时实现最小范数与最大间隔插值。
- 适用于理解Transformer基础模块的泛化机制，适合研究深度学习理论者。
良性过拟合现象——即神经网络在包含噪声的训练数据上达到完美拟合，却仍能获得接近最优的测试性能——近年来在线性模型和全连接/卷积网络中被广泛研究。本文首次研究单头softmax注意力模型中的良性过拟合，该模型是Transformer的核心组件。我们证明，在适当条件下，该模型在分类任务中仅经过两步梯度下降后即可表现出良性过拟合。此外，我们给出了最小范数/最大间隔插值器实现良性过拟合的条件。研究还揭示了过拟合行为对数据分布信噪比（SNR）的依赖性，即信号与噪声词元范数之比，并证明当且仅当信噪比足够大时，良性过拟合才可能发生。
原文摘要 · Abstract (English)
The phenomenon of benign overfitting, where a trained neural network perfectly fits noisy training data but still achieves near-optimal test performance, has been extensively studied in recent years for linear models and fully-connected/convolutional networks. In this work, we study benign overfitting in a single-head softmax attention model, which is the fundamental building block of Transformers. We prove that under appropriate conditions, the model exhibits benign overfitting in a classification setting already after two steps of gradient descent. Moreover, we show conditions where a minimum-norm/maximum-margin interpolator exhibits benign overfitting. We study how the overfitting behavior depends on the signal-to-noise ratio (SNR) of the data distribution, namely, the ratio between norms of signal and noise tokens, and prove that a sufficiently large SNR is both necessary and sufficient for benign overfitting.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。