GoLU利用偏态函数提升训练动态，性能优于GELU和Swish。
Gompertz Linear Units: Leveraging Asymmetry for Enhanced Learning Dynamics
- 基于戈姆珀兹函数的自门控激活，具右偏对称性。
- 在图像分类等6类任务中均优于GELU、Swish等先进激活函数。
- 适合追求更稳定梯度与低隐空间方差的深度学习研究者。
激活函数是深度学习架构的核心组件，显著影响训练动态。尽管ReLU应用广泛，但存在神经元死亡问题，已有LeakyReLU、PReLU、ELU等改进方案，而近期的GELU和Swish等自门控激活则通过平滑性保障稳定的梯度流并防止神经元失活。本文提出新型自门控激活函数Gompertz Linear Unit（GoLU），定义为$\mathrm{GoLU}(x) = x \, \mathrm{Gompertz}(x)$，其中$\mathrm{Gompertz}(x) = e^{-e^{-x}}$。GoLU利用戈姆珀兹函数的右偏对称性，在降低隐空间方差方面优于GELU和Swish，同时保持鲁棒的梯度流动。在图像分类、语言建模、语义分割、目标检测、实例分割及扩散模型等多个任务上的大量实验表明，GoLU在性能上超越现有主流激活函数，展现出作为替代方案的强大潜力。
原文摘要 · Abstract (English)
Activation functions are fundamental elements of deep learning architectures as they significantly influence training dynamics. ReLU, while widely used, is prone to the dying neuron problem, which has been mitigated by variants such as LeakyReLU, PReLU, and ELU that better handle negative neuron outputs. Recently, self-gated activations like GELU and Swish have emerged as state-of-the-art alternatives, leveraging their smoothness to ensure stable gradient flow and prevent neuron inactivity. In this work, we introduce the Gompertz Linear Unit (GoLU), a novel self-gated activation function defined as $\mathrm{GoLU}(x) = x \, \mathrm{Gompertz}(x)$, where $\mathrm{Gompertz}(x) = e^{-e^{-x}}$. The GoLU activation leverages the right-skewed asymmetry in the Gompertz function to reduce variance in the latent space more effectively compared to GELU and Swish, while preserving robust gradient flow. Extensive experiments across diverse tasks, including Image Classification, Language Modeling, Semantic Segmentation, Object Detection, Instance Segmentation, and Diffusion, highlight GoLU's superior performance relative to state-of-the-art activation functions, establishing GoLU as a robust alternative to existing activation functions.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。