提出MGLU结构，让语言模型推理更快更省内存。
Masked Gated Linear Unit
- 用元素级二值掩码共享权重矩阵，减少内存读取
- 在RTX5090上实现19.7倍推理加速，比标准GLU快34%
- 适合追求高效推理的LLM部署场景
门控线性单元（GLUs）是当前大语言模型前馈网络的核心组件，但其需两倍于普通前馈层的内存读取，因门控与值流使用独立权重矩阵。为解决此瓶颈，我们提出掩码门控线性单元（MGLUs），一种新型高效核实现。核心贡献包括：(1) 元素级门控混合（MoEG）架构，通过学习多个二值掩码，在单一共享权重矩阵上决定每个元素的门控或值分配，从而降低内存传输；(2) FlashMGLU，一种硬件友好型核，在RTX5090 GPU上实现较原始PyTorch MGLU高达19.7倍的推理速度提升，内存效率高47%，且比标准GLU快34%。在大语言模型实验中，采用Swish激活的SwiMGLU变体保持内存优势，并在下游任务中达到甚至超越SwiGLU基线精度。
原文摘要 · Abstract (English)
Gated Linear Units (GLUs) have become essential components in the feed-forward networks of state-of-the-art Large Language Models (LLMs). However, they require twice as many memory reads compared to feed-forward layers without gating, due to the use of separate weight matrices for the gate and value streams. To address this bottleneck, we introduce Masked Gated Linear Units (MGLUs), a novel family of GLUs with an efficient kernel implementation. The core contribution of MGLUs include: (1) the Mixture of Element-wise Gating (MoEG) architecture that learns multiple binary masks, each determining gate or value assignments at the element level on a single shared weight matrix resulting in reduced memory transfer, and (2) FlashMGLU, a hardware-friendly kernel that yields up to a 19.7 $\times$ inference-time speed-up over a naive PyTorch MGLU and is 47% more memory-efficient and 34% faster than standard GLUs despite added architectural complexity on an RTX5090 GPU. In LLM experiments, the Swish-activated variant SwiMGLU preserves its memory advantages while matching - or even surpassing - the downstream accuracy of the SwiGLU baseline.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。