提出混合注意力机制，让长文本模型更准更快。
Rope to Nope and Back Again: A New Hybrid Attention Strategy
- 融合全局与局部注意力，突破传统位置编码局限。
- 在长文本和短文本任务中均优于全注意力模型。
- 训练与推理效率显著提升，适合实际部署场景。
长上下文大语言模型虽已取得显著进展，主要得益于旋转位置编码（RoPE）及其扩展方法，但现有基于RoPE的模型在超长上下文应用中仍存在性能瓶颈。本文系统分析了包括RoPE、无位置编码（NoPE）及查询-键归一化（QK-Norm）在内的多种注意力机制，揭示其在长上下文建模中的优劣与独特注意力模式，为架构设计提供关键洞见。基于此，我们提出一种新型混合注意力架构，结合全局与局部注意力跨度。该设计不仅在长、短上下文任务上全面超越传统全注意力的RoPE模型，还在训练与推理阶段实现显著效率提升。
原文摘要 · Abstract (English)
Long-context large language models (LLMs) have achieved remarkable advancements, driven by techniques like Rotary Position Embedding (RoPE) (Su et al., 2023) and its extensions (Chen et al., 2023; Liu et al., 2024c; Peng et al., 2023). By adjusting RoPE parameters and incorporating training data with extended contexts, we can train performant models with considerably longer input sequences. However, existing RoPE-based methods exhibit performance limitations when applied to extended context lengths. This paper presents a comprehensive analysis of various attention mechanisms, including RoPE, No Positional Embedding (NoPE), and Query-Key Normalization (QK-Norm), identifying their strengths and shortcomings in long-context modeling. Our investigation identifies distinctive attention patterns in these methods and highlights their impact on long-context performance, providing valuable insights for architectural design. Building on these findings, we propose a novel architecture featuring a hybrid attention mechanism that integrates global and local attention spans. This design not only surpasses conventional RoPE-based transformer models with full attention in both long and short context tasks but also delivers substantial efficiency gains during training and inference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。