arXiv:2509.00362cs.LG2025-09

针对深度ReLU网络提出新初始化方法,解决神经元失效与梯度消失问题。

Optimized Weight Initialization on the Stiefel Manifold for Deep ReLU Neural Networks

  • 在Stiefel流形上优化正交初始化,调控前激活均值与稀疏性。
  • 理论证明可抑制神经元失效,减缓激活方差衰减,缓解梯度消失。
  • 适用于深层网络、少样本学习等场景,性能优于传统方法。

深度ReLU网络的稳定高效训练对权重初始化极为敏感。不当初始化会导致神经元永久失活(即‘死亡ReLU’)并加剧梯度不稳定性。现有方法如He、Xavier和正交初始化虽能保持方差或促进近似等距性,但未能有效调节前激活均值或控制激活稀疏性,且在极深架构中效果下降。本文提出一种专为ReLU设计的正交初始化,通过在Stiefel流形上求解优化问题,从初始阶段同时保持尺度并校准前激活统计特性。推导出一族闭式解及高效采样方案。理论分析表明,该方法能有效防止死亡ReLU现象,减缓激活方差衰减,并缓解梯度消失,从而稳定信号与梯度在深层网络中的传播。实验验证显示,在MNIST、Fashion-MNIST、多个表格数据集及少样本设置下,该方法显著优于已有初始化策略,可在极深网络中实现稳定训练。

原文摘要 · Abstract (English)

Stable and efficient training of ReLU networks with large depth is highly sensitive to weight initialization. Improper initialization can cause permanent neuron inactivation dying ReLU and exacerbate gradient instability as network depth increases. Methods such as He, Xavier, and orthogonal initialization preserve variance or promote approximate isometry. However, they do not necessarily regulate the pre-activation mean or control activation sparsity, and their effectiveness often diminishes in very deep architectures. This work introduces an orthogonal initialization specifically optimized for ReLU by solving an optimization problem on the Stiefel manifold, thereby preserving scale and calibrating the pre-activation statistics from the outset. A family of closed-form solutions and an efficient sampling scheme are derived. Theoretical analysis at initialization shows that prevention of the dying ReLU problem, slower decay of activation variance, and mitigation of gradient vanishing, which together stabilize signal and gradient flow in deep architectures. Empirically, across MNIST, Fashion-MNIST, multiple tabular datasets, few-shot settings, and ReLU-family activations, our method outperforms previous initializations and enables stable training in deep networks.

权重初始化深度学习ReLU梯度稳定

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。