用可微正则化提升神经网络权重矩阵的数值稳定性。
(Almost) Smooth Sailing: Towards Numerical Stability of Neural Networks Through Differentiable Regularization of the Condition Number
- 提出可微的条件数正则化方法，避免传统方法的不连续问题。
- 在噪声分类和MNIST去噪任务中显著提升模型稳定性和性能。
- 适用于对数值稳定性要求高的深度学习场景，如医疗或金融建模。
保持机器学习模型的数值稳定性对其可靠性与性能至关重要。一种提升网络层稳定性的方法是将权重矩阵的条件数作为正则项融入优化算法。然而，由于条件数本身不连续且不可微，难以用于梯度下降。本文提出一种新的正则化项，其几乎处处可微，并能有效促使权重矩阵具有低条件数。我们推导了该正则化项的梯度公式，易于实现并可集成至现有优化器中。实验表明，该方法在带噪声的分类任务及MNIST图像去噪中均表现优异，显著提升了模型稳定性与精度。
原文摘要 · Abstract (English)
Maintaining numerical stability in machine learning models is crucial for their reliability and performance. One approach to maintain stability of a network layer is to integrate the condition number of the weight matrix as a regularizing term into the optimization algorithm. However, due to its discontinuous nature and lack of differentiability the condition number is not suitable for a gradient descent approach. This paper introduces a novel regularizer that is provably differentiable almost everywhere and promotes matrices with low condition numbers. In particular, we derive a formula for the gradient of this regularizer which can be easily implemented and integrated into existing optimization algorithms. We show the advantages of this approach for noisy classification and denoising of MNIST images.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。