揭秘解码器仅用注意力如何感知绝对位置
Where does Absolute Position come from in decoder-only Transformers?
- 因果掩码让每个查询依赖自身绝对位置
- 残差流通过首词动态轨迹传递位置信息
- 适合研究位置编码机制的学者阅读
RoPE训练的Transformer在注意力模式中能区分绝对位置，尽管RoPE仅在内积中编码相对偏移。我们追溯这种泄漏源于两个架构组件：因果掩码因其按查询计算的softmax分母天然依赖绝对查询位置而起作用；残差流则通过首位置（0）的自闭合动力系统传播信息，下游注意力通过‘注意力吸收头’读取该轨迹。三个架构均含这两个成分，但平衡不同：NTK缩放抑制残差流成分，滑动窗口注意力使其随深度累积，标准RoPE介于两者之间。在前向传播前替换< exttt{BOS}>嵌入可消除早期查询40%的残差流成分。注意力吸收头是基于词元锚定的稳定器，当首词为预添加的< exttt{BOS}>时，其传递恒定指纹；否则随首词变化。
原文摘要 · Abstract (English)
RoPE-trained transformers distinguish absolute position in their attention patterns, even though RoPE encodes only relative offsets in the inner product. We trace this leakage to two architectural components, The causal mask is responsible for the first: its per-query softmax denominator depends on the absolute query position by construction. The residual stream supplies the second. Under causal attention the activation at position $0$ attends only to itself and runs as a closed dynamical system from the embedding of the token at that position; downstream attention reads this trajectory through sink-reading heads. Both components appear in all three architectures we study, in architecturally specific balance: NTK scaling suppresses the residual-stream component, sliding-window attention allows it to accumulate with depth, and standard RoPE sits between. Replacing the \texttt{BOS} embedding before the forward pass removes $40\%$ of the residual-stream component at early queries. Attention sinks are token-anchored stabilizers that pass forward a deterministic fingerprint of the token at position $0$, constant across inputs when that token is the auto-prepended \texttt{BOS} and varying with it otherwise.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。