GLU模型比MLP缩放更快，因具备平方阶逼近能力。
Divine Benevolence is an $x^2$: GLUs scale asymptotically faster than MLPs
- GLU通过分段二次形式实现平方阶逼近，优于MLP的线性阶。
- 在1维函数重建中，GLU的损失随参数量下降速率达P⁻³，MLP仅为P⁻²。
- 提出门控二次单元，进一步提升缩放性能，适合追求高效大模型设计者。
缩放定律可通过底层数值分析理解，传统函数逼近理论可解释模型架构选择的变化。当前前沿大模型普遍采用GLU变体，类似外积结构也在排序模型中广泛存在。这些架构的成功大多停留在经验发现层面。本文运用数值分析工具揭示关键因素：此类模型具有x²项，使其在函数逼近上实现比MLP更优的渐近缩放性能。GLU具备分段二次函数形式，足以实现二次阶逼近。核心贡献在于证明，在函数重构问题中，GLU的损失函数斜率满足L(P) ∝ P⁻³，而MLP仅为P⁻²。我们提供了参数构造与1维函数逼近的实证验证。基于此原理，我们进一步提出“门控二次单元”（Gated Quadratic Unit），其损失下降速率比GLU和MLP更陡峭。这为从第一性原理出发设计高效大模型架构提供了新路径。代码已公开于https://github.com/afqueiruga/divine_scaling。
原文摘要 · Abstract (English)
Scaling laws can be understood from ground-up numerical analysis, where traditional function approximation theory can explain shifts in model architecture choices. GLU variants now dominate frontier LLMs and similar outer-product architectures are prevalent in ranking models. The success of these architectures has mostly been left as an empirical discovery. In this paper, we apply the tools of numerical analysis to expose a key factor: these models have an $x^2$ which enables \emph{asymptotically} faster scaling than MLPs. GLUs have piecewise quadratic functional forms that are sufficient to exhibit quadratic order of approximation. Our key contribution is to demonstrate that the $L(P)$ scaling slope is $L(P)\propto P^{-3}$ for GLUs but only $L(P)=P^{-2}$ for MLPs on function reconstruction problems. We provide a parameter construction and empirical verification of these slopes for 1D function approximation. From the first principles we discover, we make one stride and propose the ``Gated Quadratic Unit'' which has an even steeper $L(P)$ slope than the GLU and MLP. This opens the possibility of architecture design from first principles numerical theory to unlock superior scaling in large models. Replication code is available at https://github.com/afqueiruga/divine_scaling.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。