arXiv:2410.19448cs.LGcs.AI2024-10

提出新指标评估梯度下降每步效率,助你提前停训省时。

Gradient Descent Efficiency Index

  • 用误差变化率与损失稳定性综合衡量每步迭代效率
  • 实验证明该指标能有效揭示梯度下降的收敛行为
  • 适合资源受限场景下优化算法选型与模型对比

梯度下降是求解多变量函数局部极小值的常用迭代算法。然而,最终迭代阶段常出现越过极小值或进展微小的问题,导致难以确定最优停止时机。本文提出一种新效率指标Ek,用于量化每一步迭代的有效性。该指标同时考虑误差的相对变化与损失函数在迭代间的稳定性。实验在多个数据集和模型上验证了Ek对梯度下降收敛行为的洞察力,可补充传统性能指标。该指数有助于机器学习中更明智地选择与调优优化算法,并用于比较不同模型的‘有效性’。

原文摘要 · Abstract (English)

Gradient descent is a widely used iterative algorithm for finding local minima in multivariate functions. However, the final iterations often either overshoot the minima or make minimal progress, making it challenging to determine an optimal stopping point. This study introduces a new efficiency metric, Ek, designed to quantify the effectiveness of each iteration. The proposed metric accounts for both the relative change in error and the stability of the loss function across iterations. This measure is particularly valuable in resource-constrained environments, where costs are closely tied to training time. Experimental validation across multiple datasets and models demonstrates that Ek provides valuable insights into the convergence behavior of gradient descent, complementing traditional performance metrics. The index has the potential to guide more informed decisions in the selection and tuning of optimization algorithms in machine learning applications and be used to compare the "effectiveness" of models relative to each other.

优化算法梯度下降训练效率

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。