通过自适应辅助变量优化深度学习，提升训练效率与稳定性。
Deep Learning Optimization Using Self-Adaptive Weighted Auxiliary Variables
- 引入自适应权重的辅助变量分离网络层，重构损失函数
- 实验验证新方法在多种场景下比梯度下降更有效且鲁棒
- 适合解决高非凸性与梯度消失问题的深度学习优化任务
本文针对全连接神经网络或物理信息神经网络的最小二乘学习问题，提出一种新的优化框架。由于损失函数高度非凸及梯度消失问题，梯度下降在深度学习中常表现低效。我们通过引入辅助变量分离网络层，并重构损失函数以简化优化过程。设计自适应权重以保持重构损失与原始均方损失的一致性，确保优化新损失即优化原问题。数值实验验证了该方法的一致性，并展示了其在多个测试场景中对梯度下降的优越性与鲁棒性。
原文摘要 · Abstract (English)
In this paper, we develop a new optimization framework for the least squares learning problem via fully connected neural networks or physics-informed neural networks. The gradient descent sometimes behaves inefficiently in deep learning because of the high non-convexity of loss functions and the vanishing gradient issue. Our idea is to introduce auxiliary variables to separate the layers of the deep neural networks and reformulate the loss functions for ease of optimization. We design the self-adaptive weights to preserve the consistency between the reformulated loss and the original mean squared loss, which guarantees that optimizing the new loss helps optimize the original problem. Numerical experiments are presented to verify the consistency and show the effectiveness and robustness of our models over gradient descent.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。