揭示高效注意力在混合模型中的真实作用，提升长文本处理能力
Rethinking the Role of Efficient Attention in Hybrid Architectures
- 用系统分析发现高效注意力影响长文本能力的演化速度
- 实验证明长距离信息由全注意力主导，高效模块优化训练路径
- 改进设计使小窗口混合模型长文本性能显著提升
现代语言模型越来越多采用混合架构，结合全注意力与高效注意力模块（如滑动窗口注意力SWA和递归序列混合器）。然而，这些高效模块如何影响模型能力仍不清晰。本文从缩放行为、机制分析和架构设计三方面进行系统研究。首先，在缩放层面发现，高效注意力设计主要影响长上下文能力的出现速度，但不同混合架构在充分训练后会收敛到相近的长文本表现。其次，机制上表明，长距离信息检索主要依赖全注意力，而高效注意力则塑造其优化轨迹。这解释了一个反直觉现象——大窗口滑动注意力会延迟全注意力层中检索头的形成。最后，基于此机制，我们发现仅对小窗口SWA混合模型的全注意力层应用NoPE，可显著提升长文本性能，且对短文本表现影响极小。
原文摘要 · Abstract (English)
Modern language models increasingly adopt hybrid architectures that combine full attention with efficient attention modules, such as sliding-window attention (SWA) and recurrent sequence mixers. However, how these efficient modules shape model capabilities remains poorly understood. To address this gap, we conduct a systematic analysis across hybrid architectures from three perspectives: scaling behavior, mechanism analysis, and architecture design. First, from a scaling perspective, we find that efficient-attention design primarily affects how fast long-context capability emerges, while different hybrids eventually converge to comparable long-context performance under sufficient training. Second, mechanistically, we show that long-range retrieval is mainly carried by full attention, whereas efficient attention shapes its optimization trajectory. This explains a counter-intuitive phenomenon we call Large-Window Laziness: larger SWA windows can delay the formation of retrieval heads in full-attention layers. Third, guided by this mechanism, we show that applying NoPE to only the full-attention layers of a small-window SWA hybrid substantially improves long-context performance with negligible impact on short-context performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。