测试百万级上下文下注意力机制是否真能解决首令牌消耗问题
Do New Attention Mechanisms Actually Fix Attention Sinks at Million-Token Context?
- 设计新检测工具SinkProbe，量化注意力集中度与位置偏差
- 发现门控机制在百万上下文下效果大幅下降，仅减少4.8%首令牌注意力
- 注意力分布、激活强度与位置偏好独立变化，说明问题复杂性更高
长上下文语言模型宣称支持百万级标记窗口，但两种习惯限制了有效使用：注意力头即使无信息也倾向于关注首个标记（称为注意力陷阱），且事实在上下文中的位置影响模型检索能力。此前研究通过门控注意力将首令牌注意力从46.7%降至4.8%，但未在百万级窗口验证。本文构建了SinkProbe测量套件，评估注意力质量、大规模激活、位置解析召回率和近期性差距，并应用于四个仅在标记混合方式与深度上不同的小模型。结果表明：训练目标导致注意力陷阱，而非架构；门控方法在本尺度下未能复现原有效果；注意力质量、激活强度与位置偏差相互独立。代码、数据与测量协议已开源。
原文摘要 · Abstract (English)
Long context language models now advertise windows of one million tokens, but two habits limit how much of that window is used. Attention heads with nothing useful to read still spend their budget on the first token, which is called the attention sink, and where a fact sits in the context changes whether the model finds it. Gated attention cut first token attention from 46.7 percent to 4.8 percent at NeurIPS 2025, and Kimi K3 pairs that idea with Kimi Delta Attention and Attention Residuals behind a one million token window, eight times past the range where these diagnostics have been reported. This paper asks whether the fix survives that jump. We build SinkProbe, a suite that measures sink mass, massive activation, position resolved recall and the recency gap, and apply it to four small models that differ only in how they mix tokens and depth. Three results follow. The training objective produces the sink, not the architecture. Gating did not reproduce its published effect at our scale. Sink mass, activations and position bias moved independently. Code, data and the measurement protocol are released at https://github.com/sararizwan7/Attention-Mechanisms-in-1M-Context-Window
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。