改进线性注意力模块，提升大模型效率与训练稳定性。
ReGLA: Refining Gated Linear Attention
- 设计新特征映射函数，解决原有方法忽略的问题。
- 引入归一化层稳定训练，缓解门控机制饱和现象。
- 在多种任务中表现优于现有线性注意力方法。
近年来大型语言模型在复杂语言任务中表现出色，但其高昂的计算与存储开销主要源于softmax注意力的二次复杂度。为缓解此问题，线性注意力被提出以降低标准Transformer固有的空间-时间复杂度。本文系统研究了门控线性注意力模块的三个关键组件：特征映射、归一化和门控机制。我们设计了一种新的特征映射函数，解决了先前方法忽视的关键问题；进一步阐明了归一化层对训练稳定的必要性；并探究了门控机制的饱和现象，引入精炼模块加以改进。通过大量实验验证，所提架构在从零训练及持续预训练后的线性化任务中均优于现有门控线性注意力方法。
原文摘要 · Abstract (English)
Recent advancements in Large Language Models (LLMs) have set themselves apart with their exceptional performance in complex language modelling tasks. However, these models are also known for their significant computational and storage requirements, primarily due to the quadratic computation complexity of softmax attention. To mitigate this issue, linear attention has been designed to reduce the quadratic space-time complexity that is inherent in standard transformers. In this work, we embarked on a comprehensive exploration of three key components that substantially impact the performance of the Gated Linear Attention module: feature maps, normalization, and the gating mechanism. We developed a feature mapping function to address some crucial issues that previous suggestions overlooked. Then we offered further rationale for the integration of normalization layers to stabilize the training process. Moreover, we explored the saturation phenomenon of the gating mechanism and augmented it with a refining module. We conducted extensive experiments and showed our architecture outperforms previous Gated Linear Attention mechanisms in extensive tasks including training from scratch and post-linearization with continual pre-training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。