VeLU通过动态调整激活响应，提升神经网络训练稳定性和性能。
VeLU: Variance-enhanced Learning Unit for Deep Neural Networks
- 基于输入方差自适应调节激活函数，结合弧函数与最优传输正则化。
- 在12个视觉基准上超越ReLU、Swish等主流激活函数，无额外参数开销。
- 适合追求训练稳定性与模型泛化能力的深度学习研究者使用。
激活函数对深层神经网络的梯度流动、优化稳定性与泛化能力至关重要。尽管ReLU因简单性被广泛使用，但存在梯度稀疏和神经元死亡问题，且对输入统计无自适应能力。平滑替代方案如Swish和GELU虽改善梯度传播，但仍采用固定变换，不随激活分布变化。本文提出VeLU，一种方差增强学习单元，通过弧正切-弧正弦变换的合理组合、自适应缩放及Wasserstein-2正则化（最优传输），实现基于局部激活方差的感知非线性。该设计使VeLU能根据激活分布动态调节响应，缓解激活层面的内部协变量偏移，提升训练稳定性，且无需增加可学习参数或结构开销。在六个深度神经网络上的大量实验表明，VeLU在12个视觉基准任务中均优于ReLU、ReLU6、Swish和GELU。代码已公开于GitHub。
原文摘要 · Abstract (English)
Activation functions play a critical role in deep neural networks by shaping gradient flow, optimization stability, and generalization. While ReLU remains widely used due to its simplicity, it suffers from gradient sparsity and dead-neuron issues and offers no adaptivity to input statistics. Smooth alternatives such as Swish and GELU improve gradient propagation but still apply a fixed transformation regardless of the activation distribution. In this paper, we propose VeLU, a Variance-enhanced Learning Unit that introduces variance-aware and distributionally aligned nonlinearity through a principled combination of ArcTan-ArcSin transformations, adaptive scaling, and Wasserstein-2 regularization (Optimal Transport). This design enables VeLU to modulate its response based on local activation variance, mitigate internal covariate shift at the activation level, and improve training stability without adding learnable parameters or architectural overhead. Extensive experiments across six deep neural networks show that VeLU outperforms ReLU, ReLU6, Swish, and GELU on 12 vision benchmarks. The implementation of VeLU is publicly available in GitHub.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。