arXiv:2601.08253cs.LG2026-01

提出新初始化方法,解决Lipschitz网络训练初期信息丢失问题。

LDLT L-Lipschitz Network Weight Parameterization Initialization

  • 基于LDLT分解设计参数初始化,精确控制输出方差。
  • 新方法使输出方差从0.41提升至0.9,缓解深层网络信息衰减。
  • 适用于需要稳定初始化的深度Lipschitz网络,如生成模型与对抗训练。

我们分析了基于LDLT的L-Lipschitz层在初始化时的动力学特性,推导出当参数矩阵$W_0\in \mathbb{R}^{m\times n}$以独立同分布高斯初始化$\mathcal{N}(0,σ^2)$时的精确边际输出方差。利用James定理和拉普拉斯积分展开$(α\boldsymbol{I}_m+S)^{-1}$,闭式推导了协方差矩阵$S=W_0W_0^\top\sim\mathcal{W}_m(n,σ^2 \boldsymbol{I}_m)$的威沙特分布。通过Isserlis/Wick组合展开,给出$\operatorname{\mathbb{E}}\left[\operatorname{tr}(S^k)\right]$的显式表达式,并计算了前10阶截断矩,对小到中等$σ^2$提供高精度级数近似。蒙特卡洛实验验证了理论估计。进一步实证显示:当前常用He或Kaiming初始化(缩放$1/\sqrt{n}$)下输出方差为0.41;而新参数化在$α=1$时使用$10/\sqrt{n}$缩放,输出方差可达0.9。结果阐明深层L-Lipschitz网络为何初始即快速丢失信息,并提供实用初始化超参数选择方案。然而,在希格斯玻色子分类数据集上的超参搜索表明,尽管理论确保方差保持,实际性能仍以He初始化为优。

原文摘要 · Abstract (English)

We analyze initialization dynamics for LDLT-based $\mathcal{L}$-Lipschitz layers by deriving the exact marginal output variance when the underlying parameter matrix $W_0\in \mathbb{R}^{m\times n}$ is initialized with IID Gaussian entries $\mathcal{N}(0,σ^2)$. The Wishart distribution, $S=W_0W_0^\top\sim\mathcal{W}_m(n,σ^2 \boldsymbol{I}_m)$, used for computing the output marginal variance is derived in closed form using expectations of zonal polynomials via James' theorem and a Laplace-integral expansion of $(α\boldsymbol{I}_m+S)^{-1}$. We develop an Isserlis/Wick-based combinatorial expansion for $\operatorname{\mathbb{E}}\left[\operatorname{tr}(S^k)\right]$ and provide explicit truncated moments up to $k=10$, which yield accurate series approximations for small-to-moderate $σ^2$. Monte Carlo experiments confirm the theoretical estimates. Furthermore, empirical analysis was performed to quantify that, using current He or Kaiming initialization with scaling $1/\sqrt{n}$, the output variance is $0.41$, whereas the new parameterization with $10/ \sqrt{n}$ for $α=1$ results in an output variance of $0.9$. The findings clarify why deep $\mathcal{L}$-Lipschitz networks suffer rapid information loss at initialization and offer practical prescriptions for choosing initialization hyperparameters to mitigate this effect. However, using the Higgs boson classification dataset, a hyperparameter sweep over optimizers, initialization scale, and depth was conducted to validate the results on real-world data, showing that although the derivation ensures variance preservation, empirical results indicate He initialization still performs better.

初始化Lipschitz深度学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。