GELU可视为随机阈值线性门的期望输出，提出新激活函数家族。
A Structural Interpretation of GELU and Threshold-Transmission Activations via the First-Order Loss Function
- 基于随机阈值的生成视角重构GELU，推导出包含ReLU等在内的门控族。
- 采用均匀分布阈值的分段多项式门在视觉与语言模型上表现优于或媲美GELU。
- 可学习的有限过渡区宽度适应不同架构，适合轻量级模型优化。
GELU通常被解释为输入相关伯努利门的期望输出。本文提出新视角：GELU是具有高斯随机阈值的硬线性门的期望输出。该解释源自随机库存理论中的经典分解，引出包含ReLU、GELU、SiLU/Swish和硬Swish的阈值传输族。通过引入潜伏的均匀阈值，恢复一种类似硬Swish的分段多项式门，其非线性转换被限定于有限区间，形成固定或可学习宽度的变体。在紧凑型视觉与语言模型上的受控实验表明，校准或可学习的均匀阈值门在性能上持续匹敌甚至超越GELU、ReLU和SiLU/Swish，表现出依赖架构的可学习宽度，并非平凡地利用有限过渡区域。
原文摘要 · Abstract (English)
The Gaussian Error Linear Unit is usually motivated as the expected output of an input-dependent Bernoulli gate. This work gives an alternative interpretation: GELU is the expected output of a hard linear gate with a Gaussian random threshold. This view provides a generative interpretation for the Bernoulli gate: the gate opens once the input clears a latent Gaussian threshold. This interpretation stems from a decomposition based on well-known results in stochastic inventory theory and leads to a threshold-transmission family that includes ReLU, GELU, SiLU/Swish, and hard swish as special cases. By considering a latent uniform threshold, we recover a hard-swish-like piecewise-polynomial gate whose nonlinear transition is confined to a finite interval, yielding fixed- and learned-width variants. Controlled experiments on compact vision and language models show that calibrated or learned uniform-threshold gates are consistently competitive with GELU, ReLU, and SiLU/Swish, display architecture-dependent learned widths, and use the finite transition region nontrivially.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。