揭示长文本注意力缩放的临界值，解释为何对数缩放能保持注意力有效性。
Critical attention scaling in long-context transformers
- 通过简化模型分析注意力缩放的相变行为
- 发现临界缩放因子为 β_n ∼ log n，过小或过大均导致失效
- 为 YaRN、Qwen 等模型提供理论支持，适合关注长文本建模的研究者
随着大语言模型上下文长度增长，注意力层出现根本性问题：注意力分数随上下文长度 n 增加而趋向均匀，导致令牌过度聚集，称为秩坍缩。虽然注意力缩放通过多项式对数因子 β_n 有效缓解此问题，但其理论依据仍不充分。本文分析一个简化的可计算模型，放大注意力缩放效应。该模型中，注意力表现出由缩放因子 β_n 控制的相变：缩放不足使所有令牌坍缩至同一方向，缩放过度则使注意力退化为恒等映射，消除令牌间有意义交互。主要结论确定临界缩放 β_n ∼ log n，为 YaRN 与 Qwen 中的对数缩放提供严格理论支撑，阐明其在长上下文下维持稀疏、内容自适应注意力的机制。
原文摘要 · Abstract (English)
As large language models scale to longer contexts, attention layers suffer from a fundamental pathology: attention scores collapse toward uniformity as context length $n$ increases, causing tokens to cluster excessively, a phenomenon known as rank-collapse. While $\textit{attention scaling}$ effectively addresses this deficiency by rescaling attention scores with a polylogarithmic factor $β_n$, theoretical justification for this approach remains lacking. We analyze a simplified yet tractable model that magnifies the effect of attention scaling. In this model, attention exhibits a phase transition governed by the scaling factor $β_n$: insufficient scaling collapses all tokens to a single direction, while excessive scaling reduces attention to identity, thereby eliminating meaningful interactions between tokens. Our main result identifies the critical scaling $β_n \asymp \log n$ and provides a rigorous justification for attention scaling in YaRN and Qwen, clarifying why logarithmic scaling maintains sparse, content-adaptive attention at large context lengths.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。