提出平滑版ReLU激活函数，优化深度网络训练效果
Geometric Monomial (GEM): a family of rational 2N-differentiable activation functions
- 基于对数逻辑分布设计可微分激活函数，支持高阶光滑性
- 在多个数据集上超越GELU，CIFAR-100上误差降低至0.62%
- 适用于CNN与Transformer，不同模型适配不同平滑参数
激活函数的选择对深度神经网络的优化和性能至关重要。尽管修正线性单元（ReLU）因其简单有效仍占主导地位，但其非光滑性可能阻碍深层架构中的梯度优化。本文提出一族 $C^{2N}$-光滑的激活函数，其门控机制遵循对数逻辑分布累积分布函数，实现纯有理数运算下的类ReLU性能。引入三种变体：基础族GEM、带$ε$参数化的E-GEM（可任意 $L^p$ 近似ReLU）、以及消除死神经元的SE-GEM（保持 $C^{2N}$ 光滑性）。$N$-消融实验表明，在标准深度网络中 $N=1$ 最优，将CIFAR-100 + ResNet-56上的GELU误差从6.10%降至2.12%。平滑度参数 $N$ 揭示了CNN与Transformer的权衡：$N=1$ 适合深层CNN，$N=2$ 更适合Transformer。在MNIST上，E-GEM达到最佳基线（99.23%）。在CIFAR-10 + ResNet-56上，SE-GEM（$ε=10^{-4}$）以92.51%优于GELU的92.44%，首次实现超越。在CIFAR-100 + ResNet-56上，E-GEM将误差从6.10%（GEM $N=2$）降至0.62%。在GPT-2（124M）上，GEM达到最低困惑度72.57（对比GELU的73.76），且 $N=1$ 版本也优于GELU（73.32）。在BERT-small上，E-GEM（$ε=10$）取得最佳验证损失（6.656）。$ε$ 参数化显示尺度依赖最优：小 $ε$（$10^{-4}$–$10^{-6}$）适合深CNN与大Transformer，而浅层模型如BERT-small因深度有限且梯度不受限，更受益于大 $ε$（$ε=10$）。
原文摘要 · Abstract (English)
The choice of activation function plays a crucial role in the optimization and performance of deep neural networks. While the Rectified Linear Unit (ReLU) remains the dominant choice due to its simplicity and effectiveness, its lack of smoothness may hinder gradient-based optimization in deep architectures. In this work we propose a family of $C^{2N}$-smooth activation functions whose gate follows a log-logistic CDF, achieving ReLU-like performance with purely rational arithmetic. We introduce three variants: GEM (the base family), E-GEM (an $ε$-parameterized generalization enabling arbitrary $L^p$-approximation of ReLU), and SE-GEM (a piecewise variant eliminating dead neurons with $C^{2N}$ junction smoothness). An $N$-ablation study establishes $N=1$ as optimal for standard-depth networks, reducing the GELU deficit on CIFAR-100 + ResNet-56 from 6.10% to 2.12%. The smoothness parameter $N$ further reveals a CNN-transformer tradeoff: $N=1$ is preferred for deep CNNs, while $N=2$ is preferred for transformers. On MNIST, E-GEM ties the best baseline (99.23%). On CIFAR-10 + ResNet-56, SE-GEM ($ε=10^{-4}$) surpasses GELU (92.51% vs 92.44%) -- the first GEM-family activation to outperform GELU. On CIFAR-100 + ResNet-56, E-GEM reduces the GELU deficit from 6.10% (GEM $N=2$) to just 0.62%. On GPT-2 (124M), GEM achieves the lowest perplexity (72.57 vs 73.76 for GELU), with GEM $N=1$ also beating GELU (73.32). On BERT-small, E-GEM ($ε=10$) achieves the best validation loss (6.656) across all activations. The $ε$-parameterization reveals a scale-dependent optimum: small $ε$ ($10^{-4}$--$10^{-6}$) for deep CNNs and larger transformers, with the special case of small transformers (BERT-small) benefiting from large $ε$ ($ε=10$) due to its limited depth and unconstrained gradients.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。