渐进式微调让大模型少动参数，省资源还高效。
Progtuning: Progressive Fine-tuning Framework for Transformer-based Language Models
- 按贡献逐步减少更新的模型层，动态分配计算资源。
- 参数更新量减少约25%，性能仍保持竞争力。
- 兼容各类高效微调方法，适配多种下游任务。
微调是利用基于Transformer的语言模型完成下游任务的有力手段。随着模型规模持续扩大，更新全部参数变得愈发昂贵。参数高效的微调方法通过仅更新少量参数有效缓解这一问题。然而，传统微调及多数参数高效微调方法需固定数量的参数更新，忽略了Transformer各模块贡献不均的问题，导致计算资源分配极不高效。本文提出Progtuning，一种结合渐进学习的新型微调框架。具体而言，Progtuning根据各Transformer模块的贡献度，逐步减少被更新的块数。显著地，Progtuning优化了资源分配，使更新参数数量减少约25%，同时保持良好性能。该方法还展现出对参数高效微调技术的高度兼容性，在多种适应场景下表现优异。
原文摘要 · Abstract (English)
Fine-tuning is a promising technique for leveraging Transformer-based language models in downstream tasks. As model sizes continue to grow, updating all model parameters becomes increasingly costly. Parameter-efficient fine-tuning methods effectively address this issue by selectively updating a small subset of parameters. However, fine-tuning and most existing parameter-efficient fine-tuning methods require updating the same number of parameters as the initial size, ignoring the unequal contribution across Transformer blocks and leading to extremely inefficient allocation of computing resources. In this paper, we propose Progtuning, the novel fine-tuning framework combined with progressive learning for Transformer-based language models. Specifically, Progtuning progressively reduces the number of updated transformer blocks based on the contribution. Remarkably, Progtuning optimizes resource allocation and reduces the number of updated parameters by approximately 25\%, while still maintaining competitive performance. And it also exhibits high adaptability with parameter-efficient fine-tuning methods, demonstrating excellent performance across various adaptation scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。