用高斯噪声替代负值，让ReLU不再有死神经。
N-ReLU: Zero-Mean Stochastic Extension of ReLU
- 负值不置零，改用高斯噪声，保持期望输出不变
- 噪声强度0.05~0.1时，准确率超或持平现有激活函数
- 无需额外参数，适合追求稳定训练的工程应用
激活函数是深度神经网络实现非线性表示的基础。然而，标准的修正线性单元（ReLU）常因硬性零截断导致神经元失活或‘死亡’。为解决此问题，我们提出N-ReLU（Noise-ReLU），一种零均值的随机扩展ReLU，将负激活值替换为高斯噪声，同时保持相同的期望输出。该期望对齐的设定使失活区域仍能维持梯度流动，并在训练中起到类似退火的正则化作用。在使用多层感知机（MLP）和卷积神经网络（CNN）架构的MNIST数据集上实验表明，当噪声标准差（sigma）为0.05~0.10时，N-ReLU的准确率与或略高于ReLU、LeakyReLU、PReLU、GELU和RReLU，且收敛稳定，未观察到死神经现象。结果表明，轻量级高斯噪声注入是一种无需修改网络结构或引入额外参数的简单而有效的优化鲁棒性增强机制。
原文摘要 · Abstract (English)
Activation functions are fundamental for enabling nonlinear representations in deep neural networks. However, the standard rectified linear unit (ReLU) often suffers from inactive or "dead" neurons caused by its hard zero cutoff. To address this issue, we introduce N-ReLU (Noise-ReLU), a zero-mean stochastic extension of ReLU that replaces negative activations with Gaussian noise while preserving the same expected output. This expectation-aligned formulation maintains gradient flow in inactive regions and acts as an annealing-style regularizer during training. Experiments on the MNIST dataset using both multilayer perceptron (MLP) and convolutional neural network (CNN) architectures show that N-ReLU achieves accuracy comparable to or slightly exceeding that of ReLU, LeakyReLU, PReLU, GELU, and RReLU at moderate noise levels (sigma = 0.05-0.10), with stable convergence and no dead neurons observed. These results demonstrate that lightweight Gaussian noise injection offers a simple yet effective mechanism to enhance optimization robustness without modifying network structures or introducing additional parameters.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。