让大模型长文本推理快3倍且不丢质量，只需简单替换注意力机制。
SWAA: Sliding Window Attention Adaptation for Efficient and Quality Preserving Long Context Processing
- 用滑动窗口注意力+分层混合策略，解决长文本信息丢失问题。
- 组合四种技巧后，长文本推理速度提升30%~100%，质量几乎不变。
- 无需重训练，直接适配已有模型，适合部署场景优化。
基于Transformer的大模型自注意力机制存在二次复杂度，导致长上下文推理成本过高。尽管滑动窗口注意力（SWA）能将复杂度降至线性，但会引发严重的性能下降，根源在于训练与推理不一致，以及结构上无法访问远距离信息。为此，我们提出滑动窗口注意力适配（SWAA），一个无需重训练的即插即用工具包。SWAA系统性结合四项核心策略：全注意力解码、交错使用全注意力与滑动窗口注意力层以缓解结构缺陷；保留“汇聚”令牌及轻量微调以缓解训练-推理差异。实验表明，单独策略效果有限，但特定组合可有效恢复长文本性能。在不同计算开销下，性能-效率权衡分析识别出最优配置，使长文本推理速度提升30%至100%，同时保持可接受的质量。代码、数据及模型权重已公开于https://github.com/yuyijiong/sliding-window-attention-adaptation。
原文摘要 · Abstract (English)
The quadratic complexity of self attention in Transformer based LLMs renders long context inference prohibitively expensive. While Sliding Window Attention (SWA), the simplest sparse attention pattern, offers a linear complexity alternative, it suffers from catastrophic long context performance collapse, which stems from two fundamental factors: the training inference mismatch when naively applying SWA to models pretrained with Full Attention (FA), and the inherent structural inability to access distant information when applying SWA to every module at all times. To address these dual challenges, we propose Sliding Window Attention Adaptation (SWAA), a plug and play toolkit of recipes that adapts FA models to SWA without costly pretraining. SWAA systematically combines four core strategies to tackle these distinct issues: (1) Full Attention (FA) Decode and (2) Interleaving FA and SWA layers, which mitigate structural defects by selectively allowing access to distant information; alongside (3) preserving ``sink'' tokens and (4) lightweight fine tuning, which mitigate the training inference mismatch. Our experiments reveal that while isolated strategies are insufficient, specific synergistic combinations effectively recover long context performance. Despite varying computational overheads, our performance efficiency trade off analysis identifies optimal SWAA configurations for diverse scenarios, achieving 30% to 100% speedups for long context inference with acceptable quality retention. Our code, data and model weights are available at https://github.com/yuyijiong/sliding-window-attention-adaptation
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。