提出一种快速近似注意力机制，让长序列预训练提速36%且不损失效果。
Multipole Semantic Attention: A Fast Approximation of Softmax Attention for Pretraining
- 分离聚类查询与键向量，生成查询专属摘要
- 64k上下文预训练提速36%，损失与基线持平
- 兼容已有模型，无需重新训练即可使用
在长序列（如整个代码库、相关文档集合）上预训练Transformer时，二次注意力开销成为瓶颈。本文提出多极语义注意力（MuSe），在不改变架构的前提下，使64k上下文预训练加速36%，同时保持与基线相当的损失。MuSe在表示空间中分别对查询和键进行聚类，生成查询特定的摘要，在相同稀疏度下显著优于空间分块方法。该方法具备即插即用特性，已在Llama 3.1-8B和3.2-1B模型上验证，无需重新训练。我们成功在代码和科学文献上对高达10亿参数的语言模型进行了64k上下文预训练，证实了MuSe在训练过程中保持了模型质量与长上下文利用能力。
原文摘要 · Abstract (English)
Pretraining transformers on long sequences (entire code repositories, collections of related documents) is bottlenecked by quadratic attention costs. We present Multipole Semantic Attention (MuSe), which accelerates 64k-context pretraining by 36% while matching baseline loss, requiring no architectural changes. MuSe clusters queries and keys separately in representation space. This yields query-specific summaries that substantially outperform spatial blocking at matched sparsity, while also enabling drop-in compatibility with existing pretrained models; we validate on Llama 3.1-8B and 3.2-1B without retraining. We pretrain language models up to 1B parameters at 64k context on code and scientific documents, confirming that MuSe preserves quality and long-context utilization during training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。