用模拟退火和共轭梯度结合训练神经网络，提升收敛性能。
SAGRAD: A Program for Neural Network Training with Simulated Annealing and the Conjugate Gradient Method
- 结合模拟退火与改进共轭梯度法优化权重更新。
- 在局部极小值或平坦区域仍能有效跳出并继续优化。
- 适用于需要稳定收敛的分类任务，适合研究者参考实现。
SAGRAD（模拟退火梯度）是一种基于Fortran 77的程序，用于通过批量学习训练神经网络进行分类。其训练过程结合了模拟退火与Møller提出的缩放共轭梯度算法——该算法是传统共轭梯度法的改进版本，更适合处理神经网络非二次性的特性。文中讨论了实现中的多个方面：包括高效计算梯度及乘以海森矩阵的向量运算；利用模拟退火重新初始化权重，以重启共轭梯度法，首次及后续每次进展不足时启用；以及在共轭梯度法取得显著进展后陷入局部极小值或权重空间平坦区时，使用模拟退火进一步探索。文章还给出了缩放共轭梯度算法、模拟退火流程及SAGRAD训练过程的概要，并展示了在两个训练数据集上的实验结果。
原文摘要 · Abstract (English)
SAGRAD (Simulated Annealing GRADient), a Fortran 77 program for computing neural networks for classification using batch learning, is discussed. Neural network training in SAGRAD is based on a combination of simulated annealing and Møller's scaled conjugate gradient algorithm, the latter a variation of the traditional conjugate gradient method, better suited for the nonquadratic nature of neural networks. Different aspects of the implementation of the training process in SAGRAD are discussed, such as the efficient computation of gradients and multiplication of vectors by Hessian matrices that are required by Møller's algorithm; the (re)initialization of weights with simulated annealing required to (re)start Møller's algorithm the first time and each time thereafter that it shows insufficient progress in reaching a possibly local minimum; and the use of simulated annealing when Møller's algorithm, after possibly making considerable progress, becomes stuck at a local minimum or flat area of weight space. Outlines of the scaled conjugate gradient algorithm, the simulated annealing procedure and the training process used in SAGRAD are presented together with results from running SAGRAD on two examples of training data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。