GLU结构通过改善模型优化速度提升性能，而非增强泛化能力。
The Devil is in the Condition Numbers: Why is GLU Better than non-GLU Structure?
- 分析两层网络在神经正切核框架下的表现，发现GLU能重塑NTK谱分布。
- GLU使条件数更小、特征值更集中，从而加快训练收敛速度。
- 适用于关注模型优化效率的研究者，尤其在大模型训练场景中。
门控线性单元（GLU）及其变体被广泛应用于现代开源大语言模型架构中，并持续优于无门控结构，但其优势背后的机制仍不明确。本文通过分析两层网络在神经正切核（NTK）框架下的行为，发现GLU结构能重塑NTK谱，使条件数更小、特征值分布更紧凑。基于此，进一步研究了训练动态，揭示了该谱形变如何导致GLU模型更快收敛，包括在损失曲线上观察到的特征性交叉现象。此外，实验证明，无论在ViT还是GPT-2上，GLU对缩小泛化差距作用有限，表明其主要优势在于加速优化过程而非提升泛化能力。代码已公开于：https://github.com/Zemdalk/GLU-NTK。
原文摘要 · Abstract (English)
Gated Linear Units (GLU) and their variants are widely adopted in modern open-source large language model architectures and consistently outperform their non-gated counterparts, yet the underlying reasons for this advantage remain unclear. In this work, we study GLU by analyzing two-layer networks in the neural tangent kernel (NTK) regime. Our analysis reveals that the GLU structure reshapes the NTK spectrum, leading to a smaller condition number and a more compact eigenvalue distribution. Building on this finding, we further analyze the resulting training dynamics and show how the reshaped spectrum leads to faster convergence of GLU models, including a characteristic loss-crossing phenomenon observed between GLU and non-GLU models. Finally, we empirically observe that GLU has limited impact in reducing the generalization gap on various models, including ViT and GPT-2, suggesting that its primary benefit lies in accelerating optimization rather than reducing the generalization gap. The code is available at: https://github.com/Zemdalk/GLU-NTK.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。