提出一种高效微调方法，仅更新极小比例参数即可达到顶尖性能。
Gradient-based Fine-Tuning through Pre-trained Model Regularization
- 基于梯度平方和选择需更新的权重行/列，提升参数选择效率
- 在FGVC和VTAB上分别仅更新1.22%和0.30%参数，显著降低存储开销
- 结合正则化增强模型迁移能力，适合资源受限场景
大规模预训练模型已在多个领域广泛应用。然而，针对特定下游任务进行微调需要大量计算资源和存储空间。现有梯度驱动参数选择（GPS）方法虽仅微调神经元中梯度高的参数，但增加了计算与存储负担。本文提出一种高效的梯度正则化微调方法（GRFT），通过更新权重矩阵的行或列实现优化。理论证明，梯度平方和最高的行/列是最优更新目标。该策略有效降低存储开销并提升参数选择效率。同时引入正则化以增强预训练模型的知识迁移能力。GRFT在多项任务上超越GPS、Adapter Tuning及LoRA等方法，在FGVC与VTAB数据集上分别仅需更新1.22%和0.30%的总参数量，展现出卓越的效率与效果。代码将尽快公开。
原文摘要 · Abstract (English)
Large pre-trained models have demonstrated extensive applications across various fields. However, fine-tuning these models for specific downstream tasks demands significant computational resources and storage. One fine-tuning method, gradient-based parameter selection (GPS), focuses on fine-tuning only the parameters with high gradients in each neuron, thereby reducing the number of training parameters. Nevertheless, this approach increases computational resource requirements and storage demands. In this paper, we propose an efficient gradient-based and regularized fine-tuning method (GRFT) that updates the rows or columns of the weight matrix. We theoretically demonstrate that the rows or columns with the highest sum of squared gradients are optimal for updating. This strategy effectively reduces storage overhead and improves the efficiency of parameter selection. Additionally, we incorporate regularization to enhance knowledge transfer from the pre-trained model. GRFT achieves state-of-the-art performance, surpassing existing methods such as GPS, Adapter Tuning, and LoRA. Notably, GRFT requires updating only 1.22% and 0.30% of the total parameters on FGVC and VTAB datasets, respectively, demonstrating its high efficiency and effectiveness. The source code will be released soon.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。