动态缩放梯度，解决微调时模型崩溃问题
Dynamic Scaled Gradient Descent for Stable Fine-Tuning for Classifications
- 按动态系数缩小正确分类样本的梯度
- 在多个数据集上降低性能波动，提升准确率
- 适合处理稀疏、不平衡数据的模型微调
微调预训练模型已成为将预训练知识适配到新稀疏、不平衡数据集以提升准确率的标准方法。然而，当优化过程陷入坍缩状态（模型停滞）时，会导致性能下降和训练不稳定。一个可能原因是训练样本间梯度相互抵消。为此，我们提出一种新算法——动态缩放梯度下降（ \mName），直接修改训练样本返回的梯度，通过动态缩放器缩小正确分类样本的梯度。该策略在理论上和实证上均提升了训练稳定性。在多种基准数据集、多个任务及大型预训练模型上的实验表明，该方法能持续降低性能方差，并超越现有方法的准确率。
原文摘要 · Abstract (English)
Fine-tuning pretrained models has become a standard approach to adapting pretrained knowledge to improve the accuracy on new sparse, imbalance datasets. However, issues arise when optimization falls into a collapsed state, where the model gets stuck, leading to degraded performance and unstable training. One possible reason for this is the cancellation of gradients across training examples. To address this problem, we propose a novel algorithm, dynamic scaled gradient descent (\mName), that directly modifies the gradients returned by training examples, specifically, scaling down the gradients of correctly classified examples using a dynamic scaler. This strategy offers both theoretical and empirical advantages in improving training stability. Experiments on a variety of benchmark datasets, spanning multiple tasks and large pretrained models, demonstrate that our method consistently reduces performance variance and surpasses the accuracy of existing approaches.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。