arXiv:2512.06427cs.LG2025-12被引 1

新初始化方法提升正弦网络梯度控制,改善训练与泛化性能。

A new initialisation to Control Gradients in Sinusoidal Neural network

  • 基于预激活分布与雅可比方差的闭式解设计初始化
  • 在函数拟合与图像重建中显著优于原有SIREN方案
  • 尤其适合物理信息神经网络等高要求重建任务

合适的初始化对缓解神经网络训练中的梯度爆炸或消失问题至关重要。然而,对于多种成熟架构,初始化参数的影响仍缺乏精确的理论理解。本文针对使用正弦激活函数的网络(如SIREN),提出一种新的初始化策略,重点在于梯度控制、其随网络深度的变化、对训练和泛化的影响。通过分析预激活分布的收敛固定点与雅可比序列的方差,推导出参数初始化的闭式表达式,该表达式区别于原始SIREN方案。该方法同时控制梯度并抑制预激活消失,防止估计过程中出现不适当频率,从而提升泛化能力。进一步通过神经切线核(NTK)框架证明该初始化强烈影响训练动态。在函数拟合与图像重建任务上,新初始化方案相比原SIREN及其它基线方法表现更优,涵盖物理信息神经网络等多种复杂重建场景。

原文摘要 · Abstract (English)

Proper initialisation strategy is of primary importance to mitigate gradient explosion or vanishing when training neural networks. Yet, the impact of initialisation parameters still lacks a precise theoretical understanding for several well-established architectures. Here, we propose a new initialisation for networks with sinusoidal activation functions such as \texttt{SIREN}, focusing on gradients control, their scaling with network depth, their impact on training and on generalization. To achieve this, we identify a closed-form expression for the initialisation of the parameters, differing from the original \texttt{SIREN} scheme. This expression is derived from fixed points obtained through the convergence of pre-activation distribution and the variance of Jacobian sequences. Controlling both gradients and targeting vanishing pre-activation helps preventing the emergence of inappropriate frequencies during estimation, thereby improving generalization. We further show that this initialisation strongly influences training dynamics through the Neural Tangent Kernel framework (NTK). Finally, we benchmark \texttt{SIREN} with the proposed initialisation against the original scheme and other baselines on function fitting and image reconstruction. The new initialisation consistently outperforms state-of-the-art methods across a wide range of reconstruction tasks, including those involving physics-informed neural networks.

正弦网络梯度控制SIREN初始化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。