滑动窗口注意力比线性注意力更高效可靠，无需后训练即可大幅降低内存消耗。
Sliding-window beats linear attention
- 采用带记忆槽的滑动窗口机制，只关注近期上下文，避免存储全部历史键值。
- 在长文本推理任务中性能比线性注意力高2到10倍，且推理速度更快。
- 无需额外训练，适合实际部署，是低成本高可靠的替代方案。
由于二次复杂度的注意力机制，大语言模型（LLMs）占用大量内存与能耗，每新增一个词元成本递增。为解决这一问题，线性注意力被提出，但尚未与更简单的基线方法充分对比。本文表明，使用记忆槽的滑动窗口注意力（SWA）在多个大模型和下游任务上表现不逊于甚至优于后训练的线性注意力模型。尤其在长上下文推理任务（如Needle-in-a-Haystack和BABILong）中，其性能高出2至10倍。SWA无需后训练，计算极快，内存开销极低，是极具性价比的解决方案。因此，我们强烈建议用SWA替代后训练线性注意力模型。线性注意力可能仍具潜力，但需从头训练或大规模微调才能达到类似效果。
原文摘要 · Abstract (English)
Due to the nature of quadratic attention, Large Language Models (LLMs) consume a lot of memory and energy. Every new token costs more than the previous one. For each additional token, the keys and values must be stored in memory indefinitely, which is unsustainable. Several alternatives have been proposed to fix the quadratic scaling problem, one of which is retrofitting LLMs to use Linear Attention. This idea has attracted a lot of attention, given its promise to solve the quadratic scaling problem with state-of-the-art performance at low cost. However, this line of research has not been properly compared to simpler baselines. In this work, we show that Sliding Window Attention (SWA) with sinks performs as well or better than post-trained Linear Attention models. We observe this across multiple LLMs on various downstream tasks. For long-context reasoning tasks (Needle-in-a-Haystack and BABILong), SWA achieves massively higher performance (2 to 10 times higher than linear attention). SWA requires no post-training, is extremely fast, and requires low memory; therefore, making it an extremely cheap and reliable solution. To reduce inference memory cost, we strongly recommend switching to SWA instead of post-training linear models. Linear attention models may have shown some promise, but they likely require to be trained from scratch or extensive post-training in order to even match SWA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。