提出分层移位混合法，让Transformer计算更高效且不损失性能。
Hierarchical Shift Mixing -- Beyond Dense Attention in Transformers
- 将令牌间交互分散到多层处理，避免每层都密集计算。
- 实现线性时间复杂度，简单版本性能接近原版注意力。
- 适合追求高效推理与训练的NLP模型开发者使用。
自大语言模型引入Transformer架构以来，基于Softmax的注意力层因其二次时间复杂度受到越来越多质疑。尽管已有研究尝试用更低复杂度的方法替代它，但多数情况下会降低性能。本文提出分层移位混合法（Hierarchical Shift Mixing, HSM），一种通用的令牌混合框架，将成对令牌交互分布到Transformer各层中，而非在每层内密集计算。HSM实现了线性时间复杂度，且对具体混合函数无偏好。我们证明，即使简单的HSM变体也能达到接近Softmax注意力的性能；而将HSM与Softmax注意力结合的混合架构，在训练和推理阶段均显著降低计算开销的同时，仍优于标准GPT型Transformer基线。
原文摘要 · Abstract (English)
Since the introduction of the Transformer architecture for large language models, the softmax-based attention layer has faced increasing scrutinity due to its quadratic-time computational complexity. Attempts have been made to replace it with less complex methods, at the cost of reduced performance in most cases. We introduce Hierarchical Shift Mixing (HSM), a general framework for token mixing that distributes pairwise token interactions across Transformer layers rather than computing them densely within each layer. HSM enables linear-time complexity while remaining agnostic to the specific mixing function. We show that even simple HSM variants achieve performance close to softmax attention, and that hybrid architectures combining HSM with softmax attention can outperform a GPT-style Transformer baseline while reducing computational cost during both training and inference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。