通过变换损失函数提升梯度敏感性，加速非凸优化收敛
AYLA: Amplifying Gradient Sensitivity via Loss Transformation in Non-Convex Optimization
- 对损失函数施加可调幂律变换，增强梯度响应
- 在多项式、曲线拟合及图像分类任务中均快于SGD和ADAM
- 无需修改模型即可提升现有优化器性能，适合深度学习训练
随机梯度下降（SGD）及其变体如ADAM是深度学习优化的基础，通过固定或自适应学习率调整模型参数。但在高维非凸场景下，这些方法常难以兼顾适应性与效率。本文提出AYLA，一种通过损失函数变换增强训练动态的新框架。该方法对损失施加可调幂律变换，在保持极值点不变的同时放大损失值，从而提升梯度敏感性并加速收敛。此外，我们设计了一种随变换后损失动态调整的学习率，进一步提高优化效率。在最小化一个合成非凸多项式、求解非凸曲线拟合任务，以及进行手写数字分类（MNIST）和图像识别（CIFAR-100）的实验中，AYLA在收敛速度和训练稳定性上均持续优于SGD和ADAM。通过重塑损失景观，AYLA为现有优化方法提供了无需模型修改的通用增强，为深度神经网络训练带来新进展。
原文摘要 · Abstract (English)
Stochastic Gradient Descent (SGD) and its variants, such as ADAM, are foundational to deep learning optimization, adjusting model parameters through fixed or adaptive learning rates based on loss function gradients. However, these methods often struggle to balance adaptability and efficiency in high-dimensional, non-convex settings. This paper introduces AYLA, a novel optimization framework that enhances training dynamics via loss function transformation. AYLA applies a tunable power-law transformation to the loss, preserving critical points while scaling loss values to amplify gradient sensitivity and accelerate convergence. Additionally, we propose an effective learning rate that dynamically adapts to the transformed loss, further improving optimization efficiency. Empirical evaluations on minimizing a synthetic non-convex polynomial, solving a non-convex curve-fitting task, and performing digit classification (MNIST) and image recognition (CIFAR-100) demonstrate that AYLA consistently outperforms SGD and ADAM in both convergence speed and training stability. By reshaping the loss landscape, AYLA provides a model-agnostic enhancement to existing optimization methods, offering a promising advancement in deep neural network training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。