用滑动窗口注意力训练，让大模型高效处理长文本。
Sliding Window Attention Training for Efficient Large Language Models
- 用sigmoid替代softmax，缓解注意力坍缩问题。
- 在8个基准上达到当前最优性能，优于线性递归结构。
- 保持Transformer原结构，适合追求效率的长文本场景。
基于Transformer的大语言模型在多项任务中表现出色，但其序列长度相关的二次计算复杂度仍是处理长文档的主要瓶颈。为此，许多研究提出稀疏注意力和状态空间模型等方法以提升长序列效率，但往往牺牲性能或引入结构复杂性。本文提出SWAT，通过滑动窗口注意力训练实现高效长上下文处理。首先指出Transformer效率低下的根源在于softmax操作带来的高方差导致的注意力坍缩现象；随后采用sigmoid函数替代softmax，并结合平衡的ALiBi与旋转位置编码，实现信息的高效压缩与保留。实验表明，SWAT在8个基准测试中达到当前最优性能，优于最先进的线性递归架构。代码已开源：https://github.com/Fzkuji/swat-attention。
原文摘要 · Abstract (English)
Recent advances in transformer-based Large Language Models (LLMs) have demonstrated remarkable capabilities across various tasks. However, their quadratic computational complexity concerning sequence length remains a significant bottleneck for processing long documents. As a result, many efforts like sparse attention and state space models have been proposed to improve the efficiency of LLMs over long sequences. Though effective, these approaches compromise the performance or introduce structural complexity. This calls for a simple yet efficient model that preserves the fundamental Transformer architecture. To this end, we introduce SWAT, which enables efficient long-context handling via Sliding Window Attention Training. This paper first attributes the inefficiency of Transformers to the attention sink phenomenon resulting from the high variance of softmax operation. Then, we replace softmax with the sigmoid function and utilize a balanced ALiBi and Rotary Position Embedding for efficient information compression and retention. Experiments demonstrate that SWAT achieves SOTA performance compared with state-of-the-art linear recurrent architectures on eight benchmarks. Code is available at https://github.com/Fzkuji/swat-attention.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。