针对tanh网络提出新初始化方法，提升深度网络训练稳定性与收敛速度。
Robust Weight Initialization for Tanh Neural Networks with Fixed Point Analysis
- 基于tanh(ax)的不动点分析，优化参数a以缓解激活饱和
- 在多个数据集上优于Xavier初始化，尤其在大模型和小样本下表现更优
- 适合追求稳定训练的深度tanh网络研究者或工程应用
随着神经网络深度增加，泛化性能通常会提升，但训练深网络面临梯度消失与信号传播问题。尽管已有大量理论研究和方法，针对tanh激活函数的高效权重初始化仍缺乏深入探索。本文提出一种新的tanh神经网络权重初始化方法，基于对函数$ anh(ax)$不动点的分析，旨在确定使激活函数避免饱和的参数值$a$。在多种分类数据集及物理信息神经网络上的实验表明，该方法在不同网络规模、数据效率和收敛速度方面均优于带或不带归一化的Xavier初始化。代码已开源：https://github.com/1HyunwooLee/Tanh-Init。
原文摘要 · Abstract (English)
As a neural network's depth increases, it can improve generalization performance. However, training deep networks is challenging due to gradient and signal propagation issues. To address these challenges, extensive theoretical research and various methods have been introduced. Despite these advances, effective weight initialization methods for tanh neural networks remain insufficiently investigated. This paper presents a novel weight initialization method for neural networks with tanh activation function. Based on an analysis of the fixed points of the function $\tanh(ax)$, the proposed method aims to determine values of $a$ that mitigate activation saturation. A series of experiments on various classification datasets and physics-informed neural networks demonstrates that the proposed method outperforms Xavier initialization methods~(with or without normalization) in terms of robustness across different network sizes, data efficiency, and convergence speed. Code is available at https://github.com/1HyunwooLee/Tanh-Init
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。