提出平滑权重矩阵奇异谱的方法，解决深度网络训练中的不稳定性问题。
Dispelling the Curse of Singularities in Neural Network Optimizations
- 通过分析参数空间奇异性的增长机制，揭示其与表示空间的相互强化
- 实验显示该方法可缓解损失突增，提升训练成功率与泛化性能
- 适合追求训练稳定性和模型鲁棒性的研究者和工程师
本文从参数空间中奇异性出现与放大的视角，探讨深度神经网络的优化不稳定性。分析表明，参数奇异性随梯度更新不可避免地增长，并进一步增强与表示空间的对齐，导致表示空间奇异性加剧。我们证明梯度Frobenius范数受权重矩阵最大奇异值约束，而训练过程中权重与表示奇异性相互强化，形成‘奇异性诅咒’，松弛了该约束，增加了尖锐损失爆炸的风险。为此，我们提出轻量、灵活且高效的参数奇异性平滑（PSS）方法，用于平滑权重矩阵的奇异谱。在多种数据集、架构和优化器上的广泛实验表明，PSS能有效缓解不稳定性，恢复失败后的可训练性，并提升训练效率与泛化能力。
原文摘要 · Abstract (English)
This work investigates the optimization instability of deep neural networks from a less-explored yet insightful perspective: the emergence and amplification of singularities in the parametric space. Our analysis reveals that parametric singularities inevitably grow with gradient updates and further intensify alignment with representations, leading to increased singularities in the representation space. We show that the gradient Frobenius norms are bounded by the top singular values of the weight matrices, and as training progresses, the mutually reinforcing growth of weight and representation singularities, termed the curse of singularities, relaxes these bounds, escalating the risk of sharp loss explosions. To counter this, we propose Parametric Singularity Smoothing (PSS), a lightweight, flexible, and effective method for smoothing the singular spectra of weight matrices. Extensive experiments across diverse datasets, architectures, and optimizers demonstrate that PSS mitigates instability, restores trainability even after failure, and improves both training efficiency and generalization.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。