AdamZ动态调节学习率，解决训练中过冲和停滞问题。
AdamZ: An Enhanced Optimisation Method for Neural Network Training
- 根据过冲与停滞情况自适应调整学习率
- 在多种任务上显著降低损失函数值
- 适合对精度要求高的场景，如科学计算
AdamZ 是 Adam 优化器的一种改进版本，旨在提升神经网络训练中的收敛效率。该优化器通过动态调整学习率来应对训练中常见的过冲和停滞问题：检测到过冲时降低学习率，停滞时则提高学习率。其调节机制依赖于过冲因子、停滞因子、阈值及耐心水平等超参数。尽管相较于部分优化器训练时间稍长，但 AdamZ 在最小化损失函数方面表现稳定且优异，适用于对精度要求较高的应用。基准测试表明，该方法能有效维持最优学习率，显著提升模型在多样化任务上的性能。
原文摘要 · Abstract (English)
AdamZ is an advanced variant of the Adam optimiser, developed to enhance convergence efficiency in neural network training. This optimiser dynamically adjusts the learning rate by incorporating mechanisms to address overshooting and stagnation, that are common challenges in optimisation. Specifically, AdamZ reduces the learning rate when overshooting is detected and increases it during periods of stagnation, utilising hyperparameters such as overshoot and stagnation factors, thresholds, and patience levels to guide these adjustments. While AdamZ may lead to slightly longer training times compared to some other optimisers, it consistently excels in minimising the loss function, making it particularly advantageous for applications where precision is critical. Benchmarking results demonstrate the effectiveness of AdamZ in maintaining optimal learning rates, leading to improved model performance across diverse tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。