证明注意力层能精准定位单个关键信息点并回归输出。
Attention layers provably solve single-location regression
- 用单位置回归任务验证注意力机制的定位能力。
- 模型在非凸优化下仍能收敛到最优解，实现渐近贝叶斯最优。
- 适合研究注意力机制理论性质或稀疏信号建模的研究者。
基于注意力的模型（如Transformer）在各类任务中表现优异，但对其局部稀疏性与内部线性表示的理论理解仍不充分。为此，本文提出单位置回归任务：序列中仅一个位置的标记决定输出，其位置为隐变量，可通过输入的线性投影恢复。我们设计了一个专用预测器，实为简化版非线性自注意力层。通过分析其渐近贝叶斯最优性及训练动态，发现尽管问题非凸，该预测器仍能有效学习底层结构。本工作揭示了注意力机制处理稀疏标记信息与内部线性结构的能力。
原文摘要 · Abstract (English)
Attention-based models, such as Transformer, excel across various tasks but lack a comprehensive theoretical understanding, especially regarding token-wise sparsity and internal linear representations. To address this gap, we introduce the single-location regression task, where only one token in a sequence determines the output, and its position is a latent random variable, retrievable via a linear projection of the input. To solve this task, we propose a dedicated predictor, which turns out to be a simplified version of a non-linear self-attention layer. We study its theoretical properties, by showing its asymptotic Bayes optimality and analyzing its training dynamics. In particular, despite the non-convex nature of the problem, the predictor effectively learns the underlying structure. This work highlights the capacity of attention mechanisms to handle sparse token information and internal linear structures.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。