SWAN-GPT让大模型轻松处理超长文本，无需额外训练
SWAN-GPT: An Efficient and Scalable Approach for Long-Context Language Modeling
- 用无位置编码与滑动窗口注意力混合设计，突破训练长度限制
- 推理时动态缩放注意力分数，支持远超训练长度的上下文
- 效率更高，旧模型可快速改造为长序列版本
我们提出一种解码器仅有的Transformer架构——SWAN-GPT，能稳健泛化到远超训练时长度的序列。该模型通过交替使用无位置编码（NoPE）层和带旋转位置编码（SWA-RoPE）的滑动窗口注意力层实现。实验表明，无需额外长上下文训练即可在显著更长的序列上保持优异性能。这一鲁棒的长度外推能力源于新颖的架构设计，并辅以推理阶段简单的注意力分数动态缩放机制。此外，SWAN-GPT比标准GPT架构更具计算效率，训练成本更低，吞吐量更高。我们还证明，现有预训练的解码器仅模型可通过极少微调快速转换为SWAN架构，从而支持更长上下文。整体而言，本工作提供了一种高效且稳健的长上下文语言建模扩展方案。
原文摘要 · Abstract (English)
We present a decoder-only Transformer architecture that robustly generalizes to sequence lengths substantially longer than those seen during training. Our model, SWAN-GPT, interleaves layers without positional encodings (NoPE) and sliding-window attention layers equipped with rotary positional encodings (SWA-RoPE). Experiments demonstrate strong performance on sequence lengths significantly longer than the training length without the need for additional long-context training. This robust length extrapolation is achieved through our novel architecture, enhanced by a straightforward dynamic scaling of attention scores during inference. In addition, SWAN-GPT is more computationally efficient than standard GPT architectures, resulting in cheaper training and higher throughput. Further, we demonstrate that existing pre-trained decoder-only models can be efficiently converted to the SWAN architecture with minimal continued training, enabling longer contexts. Overall, our work presents an effective approach for scaling language models to longer contexts in a robust and efficient manner.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。