揭示注意力机制中位置编码的隐藏训练偏差
Deconstructing Positional Information: From Attention Logits to Training Biases
- 区分加法与乘法位置编码，解析其捕捉位置信息的机制差异
- 在需融合语义与位置信息的任务中，乘法编码显著更优
- 发现浅层网络存在单头信息堆积现象，源于乘法编码本质
位置编码使Transformer能够融入序列信息，但其理论理解仍局限于距离衰减和平移不变性两个特性。由于自然语言缺乏纯粹的位置数据，位置与语义信息的相互作用尚未深入探究。本文通过分解注意力对数计算，系统分析位置编码，将其分为加法与乘法形式。二者性质不同，导致捕捉位置信息的机制各异。为验证差异，设计了一项显式要求位置与语义深度融合的合成任务。结果表明，乘法编码在此任务上表现显著更优。此外，评估揭示了一个隐藏的训练偏差：浅层中存在信息聚合效应，称为单头沉积模式。通过消融实验与理论分析证实，该现象源于乘法编码的本质。研究深化了对位置编码的理解，并呼吁进一步探索其训练动态。
原文摘要 · Abstract (English)
Positional encodings enable Transformers to incorporate sequential information, yet their theoretical understanding remains limited to two properties: distance attenuation and translation invariance. Because natural language lacks purely positional data, the interplay between positional and semantic information is still underexplored. We address this gap by deconstructing the attention-logit computation and providing a structured analysis of positional encodings, categorizing them into additive and multiplicative forms. The differing properties of these forms lead to distinct mechanisms for capturing positional information. To probe this difference, we design a synthetic task that explicitly requires strong integration of positional and semantic cues. As predicted, multiplicative encodings achieve a clear performance advantage on this task. Moreover, our evaluation reveals a hidden training bias: an information aggregation effect in shallow layers that we term the single-head deposit pattern. Through ablation studies and theoretical analysis, we proved that this phenomenon is inherent in multiplicative encodings. These findings deepen the understanding of positional encodings and call for further study of their training dynamics.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。