用SVD预条件化改进梯度下降，让非线性拟合更快更准。
SVD-Preconditioned Gradient Descent Method for Solving Nonlinear Least Squares Problems
- 用雅可比矩阵的SVD构造预条件器，动态调整梯度方向。
- 在函数逼近、PDE求解和CIFAR-10上均比Adam收敛更快、误差更低。
- 适合需要高精度与快速收敛的非线性优化任务，如科学计算与图像识别。
本文提出一种针对非线性最小二乘问题的新优化算法。该方法通过使用雅可比矩阵的奇异值分解（SVD）对梯度下降方向进行预条件化，并将其与Adam优化器的一阶和二阶动量自适应学习率机制相结合。在标准正则性假设下，我们建立了该方法的局部线性收敛性，并在适当条件下证明了其变体的全局收敛性。实验结果表明，该方法在函数逼近、偏微分方程求解以及在CIFAR-10数据集上的图像分类任务中均表现出色，显著优于标准Adam，实现了更快的收敛速度和更低的误差。
原文摘要 · Abstract (English)
This paper introduces a novel optimization algorithm designed for nonlinear least-squares problems. The method is derived by preconditioning the gradient descent direction using the Singular Value Decomposition (SVD) of the Jacobian. This SVD-based preconditioner is then integrated with the first- and second-moment adaptive learning rate mechanism of the Adam optimizer. We establish the local linear convergence of the proposed method under standard regularity assumptions and prove global convergence for a modified version of the algorithm under suitable conditions. The effectiveness of the approach is demonstrated experimentally across a range of tasks, including function approximation, partial differential equation (PDE) solving, and image classification on the CIFAR-10 dataset. Results show that the proposed method consistently outperforms standard Adam, achieving faster convergence and lower error in both regression and classification settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。