发现层归一化导致Transformer解码器产生近期偏好
LayerNorm Induces Recency Bias in Transformer Decoders
- 层归一化与因果自注意力共同引发对最近词元的偏好
- 仅靠因果自注意力会产生早期偏好，与实际观察不符
- 研究为改进位置编码提供理论依据，适合模型设计者
因果自注意力为Transformer解码器提供位置信息。已有研究表明，仅堆叠因果自注意力层会引发注意力分数对早期词元的偏好。然而，实际中观察到的通常是向近期词元偏移的“近期偏差”。本文通过分析因果自注意力与其他结构组件的交互，发现堆叠的因果自注意力结合层归一化（LayerNorm）会诱导出近期偏差。此外，我们还考察了残差连接和输入词元嵌入分布对此偏差的影响。结果揭示了位置信息如何与架构组件相互作用，为改进位置编码策略提供了新的理论视角。
原文摘要 · Abstract (English)
Causal self-attention provides positional information to Transformer decoders. Prior work has shown that stacks of causal self-attention layers alone induce a positional bias in attention scores toward earlier tokens. However, this differs from the bias toward later tokens typically observed in Transformer decoders, known as recency bias. We address this discrepancy by analyzing the interaction between causal self-attention and other architectural components. We show that stacked causal self-attention layers combined with LayerNorm induce recency bias. Furthermore, we examine the effects of residual connections and the distribution of input token embeddings on this bias. Our results provide new theoretical insights into how positional information interacts with architectural components and suggest directions for improving positional encoding strategies.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。