用门控线性单元改进注意力机制，提升模型性能与收敛速度。
GLU Attention Improve Transformer
- 在注意力值中引入非线性，通过门控线性单元增强表达能力。
- 跨文本与视觉任务均提升性能，且收敛更快，零新增参数。
- 轻量设计，兼容Flash Attention、RoPE等主流技术，适合部署优化。
门控线性单元（GLU）在提升神经网络性能方面展现出巨大潜力。本文提出一种新型注意力机制——GLU Attention，将非线性引入注意力的值部分。实验表明，该方法在文本与视觉模态上均显著提升模型性能与收敛速度，且不增加额外参数，计算开销可忽略。GLU Attention 轻量化设计，可无缝集成于 Flash Attention、旋转位置编码（RoPE）以及多种多头注意力变体（如分组查询注意力 GQA）。相关代码已开源至 GitHub。
原文摘要 · Abstract (English)
Gated Linear Units (GLU) have shown great potential in enhancing neural network performance. In this paper, I introduce a novel attention mechanism called GLU Attention, which introduces nonlinearity into the values of Attention. My experiments demonstrate that GLU Attention improves both model performance and convergence speed across text and vision modalities with zero additional parameters and negligible computational costs. GLU Attention is lightweight and can seamlessly integrate with other technologies, such as Flash Attention, Rotary Position Embedding (RoPE), and various Multi-Head Attention (MHA) variants such as Grouped-Query Attention (GQA). This project is open-sourced at github.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。