提出可学习的动态激活函数，让神经网络训练更快更稳。
dynActivation: A Trainable Activation Family for Adaptive Nonlinearity
- 用可学习参数动态调节激活函数，介于非线性和线性之间。
- 在图像和语言任务中，比ReLU快54%，精度提升最高14%。
- 适合追求训练效率与模型稳定性的深度学习研究者。
本文提出dynActivation，一种每层可学习的激活函数，形式为 $f_i(x) = \mathrm{BaseAct}(x)(α_i - β_i) + β_i x$，其中 $α_i$ 和 $β_i$ 是轻量级可学习标量，用于在基础非线性与线性路径间插值，$\mathrm{BaseAct}(x)$ 类似ReLU。在多个视觉任务、语言建模任务及消融实验中对比了静态与动态ReLU类变体。结果表明，dynActivation变体倾向于使深层网络线性化，同时保持高性能，相比ReLU训练效率最高提升54%。在CIFAR-10上，dynActivation(Mish)在AttentionCNN上最高提升14.02%，平均提升6.00%，收敛AUC降低24%（2120 vs. 2785）。在1到75层的MNIST深度扩展实验中，dynActivation测试准确率始终不低于95%（95.3–99.3%），而ReLU在25层时降至80%以下。在FGSM攻击下（ε=0.08），dynActivation(Mish)精度下降55.39%，优于ReLU的62.79%（优势7.40%）。迁移至语言建模，新提出的dynActGLU变体在5620步时相对困惑度降低10.3%（4.047 vs. 4.514），但34300步后差距消失。
原文摘要 · Abstract (English)
This paper proposes $\mathrm{dynActivation}$, a per-layer trainable activation defined as $f_i(x) = \mathrm{BaseAct}(x)(α_i - β_i) + β_i x$, where $α_i$ and $β_i$ are lightweight learned scalars that interpolate between the base nonlinearity and a linear path and $\mathrm{BaseAct}(x)$ resembles any ReLU-like function. The static and dynamic ReLU-like variants are then compared across multiple vision tasks, language modeling tasks, and ablation studies. The results suggest that dynActivation variants tend to linearize deep layers while maintaining high performance, which can improve training efficiency by up to $+54\%$ over ReLU. On CIFAR-10, dynActivation(Mish) improves over static Mish by up to $+14.02\%$ on AttentionCNN with an average improvment by $+6.00\%$, with a $24\%$ convergence-AUC reduction relative to Mish (2120 vs. 2785). In a 1-to-75-layer MNIST depth-scaling study, dynActivation never drops below $95\%$ test accuracy ($95.3$--$99.3\%$), while ReLU collapses below $80\%$ at 25 layers. Under FGSM at $\varepsilon{=}0.08$, dynActivation(Mish) incurs a $55.39\%$ accuracy drop versus $62.79\%$ for ReLU ($7.40\%$ advantage). Transferred to language modeling, a new proposed dynActGLU-variant achieves a $10.3\%$ relative perplexity reduction over SwiGLU at 5620 steps (4.047 vs. 4.514), though the gap vanishes at 34300 steps.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。