通过调整权重矩阵奇异值，让神经网络训练更稳定高效。
Weight Conditioning for Smooth Optimization of Neural Networks
- 通过压缩权重矩阵奇异值范围，改善矩阵条件数。
- 在多种模型上验证，显著提升梯度下降收敛速度。
- 适合追求训练稳定性和加速的深度学习研究者。
本文提出一种新型神经网络权重矩阵归一化技术——权重条件化（Weight Conditioning），旨在缩小权重矩阵最小与最大奇异值之间的差距，从而获得条件更好的矩阵。该方法受数值线性代数启发：良好条件的矩阵能为迭代求解器提供更强的收敛保证。我们提供了理论依据，证明该归一化可平滑损失曲面，进而提升随机梯度下降算法的收敛性能。实验验证了该方法在多种架构上的有效性，包括卷积神经网络（CNN）、视觉变换器（ViT）、神经辐射场（NeRF）以及3D形状建模任务中，结果表明其不仅具有竞争力，且优于现有权重归一化方法。
原文摘要 · Abstract (English)
In this article, we introduce a novel normalization technique for neural network weight matrices, which we term weight conditioning. This approach aims to narrow the gap between the smallest and largest singular values of the weight matrices, resulting in better-conditioned matrices. The inspiration for this technique partially derives from numerical linear algebra, where well-conditioned matrices are known to facilitate stronger convergence results for iterative solvers. We provide a theoretical foundation demonstrating that our normalization technique smoothens the loss landscape, thereby enhancing convergence of stochastic gradient descent algorithms. Empirically, we validate our normalization across various neural network architectures, including Convolutional Neural Networks (CNNs), Vision Transformers (ViT), Neural Radiance Fields (NeRF), and 3D shape modeling. Our findings indicate that our normalization method is not only competitive but also outperforms existing weight normalization techniques from the literature.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。