基于海森矩阵信息动态调整学习率，提升训练收敛速度。
A Hessian-informed hyperparameter optimization for differential learning rate
- 用海森矩阵信息指导学习率分组优化，自适应捕捉损失曲率。
- 实验表明该方法能显著加快收敛，无需手动调参。
- 适合需要高效微调的深度模型，尤其对参数量大的场景有效。
差异学习率（DLR）是一种为不同模型参数分配不同学习率的技术，在深度学习中广泛应用并取得良好效果。例如，参数高效微调（PEFT）将多数参数的学习率设为零，大幅降低计算成本。其核心思想是：不同参数具有不同的损失曲率，但通常难以准确刻画。本文提出海森矩阵感知的差异学习率（Hi-DLR），一种高效的超参数优化方法，可自适应地为任意模型和优化器确定学习率，捕捉损失曲率。在合理参数分组下，实验证明Hi-DLR能通过训练过程中的动态学习率调整显著提升收敛性。
原文摘要 · Abstract (English)
Differential learning rate (DLR), a technique that applies different learning rates to different model parameters, has been widely used in deep learning and achieved empirical success via its various forms. For example, parameter-efficient fine-tuning (PEFT) applies zero learning rates to most parameters so as to significantly save the computational cost. At the core, DLR leverages the observation that different parameters can have different loss curvature, which is hard to characterize in general. We propose the Hessian-informed differential learning rate (Hi-DLR), an efficient approach that solves the hyperparameter optimization (HPO) of learning rates and captures the loss curvature for any model and optimizer adaptively. Given a proper grouping of parameters, we empirically demonstrate that Hi-DLR can improve the convergence by dynamically determining the learning rates during the training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。