arXiv:2507.22090cs.LGcs.AI2025-07被引 1

提出S4激活函数,解决梯度消失与死神经问题,提升训练稳定性。

Hybrid activation functions for deep neural networks: S3 and S4 -- a novel approach to gradient flow optimization

  • S4融合sigmoid与softsign,通过可调参数实现平滑过渡。
  • 在MNIST上达97.4%准确率,收敛速度比ReLU快19%。
  • 适合对梯度稳定性要求高的深度模型,如医疗或金融预测。

激活函数是深度神经网络的关键组件,直接影响梯度流动、训练稳定性和模型性能。传统函数如ReLU存在死神经问题,而sigmoid和tanh则有梯度消失缺陷。本文提出两种新型混合激活函数:S3(Sigmoid-Softsign)及其改进版S4(平滑S3)。S3对负输入使用sigmoid,正输入使用softsign;S4引入陡度参数k控制平滑过渡。在三种网络架构上,针对二分类、多分类与回归任务进行实验。S4优于九种基线函数,在MNIST上达97.4%准确率,Iris分类96.0%,波士顿房价回归18.7 MSE。其收敛速度比ReLU快19%,深层网络中梯度范围为[0.24, 0.59],死神经比例仅18%(相比ReLU)。可调参数k使S4适配不同任务与深度,展现强大泛化能力。结果表明,混合激活函数是优化训练动态的可行方向。

原文摘要 · Abstract (English)

Activation functions are critical components in deep neural networks, directly influencing gradient flow, training stability, and model performance. Traditional functions like ReLU suffer from dead neuron problems, while sigmoid and tanh exhibit vanishing gradient issues. We introduce two novel hybrid activation functions: S3 (Sigmoid-Softsign) and its improved version S4 (smoothed S3). S3 combines sigmoid for negative inputs with softsign for positive inputs, while S4 employs a smooth transition mechanism controlled by a steepness parameter k. We conducted comprehensive experiments across binary classification, multi-class classification, and regression tasks using three different neural network architectures. S4 demonstrated superior performance compared to nine baseline activation functions, achieving 97.4% accuracy on MNIST, 96.0% on Iris classification, and 18.7 MSE on Boston Housing regression. The function exhibited faster convergence (-19 for ReLU) and maintained stable gradient flow across network depths. Comparative analysis revealed S4's gradient range of [0.24, 0.59] compared to ReLU's 18% dead neurons in deep networks. The S4 activation function addresses key limitations of existing functions through its hybrid design and smooth transition mechanism. The tunable parameter k allows adaptation to different tasks and network depths, making S4 a versatile choice for deep learning applications. These findings suggest that hybrid activation functions represent a promising direction for improving neural network training dynamics.

激活函数梯度优化深度学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。