只训练关键适配器，大幅降低大模型微调的内存与计算开销。
Not All Adapters Matter: Selective Adapter Freezing for Memory-Efficient Fine-Tuning of Language Models
- 按重要性动态冻结低贡献适配器，减少冗余计算。
- 内存、计算量和训练时间分别降低42.85%、34.59%、11.82%。
- 可提升泛化能力，适合资源受限场景下的高效微调。
基于Transformer的大规模预训练模型取得了巨大成功，微调是其在下游任务中应用的标准方法。其中，适配器微调通过引入轻量级可训练模块，在保持大部分预训练参数冻结的同时实现参数高效微调。然而，现有适配器微调方法仍需大量资源。我们发现，每个适配器对任务性能和资源消耗的贡献不均。受此启发，我们提出选择性适配器冻结（SAFE），早期逐步冻结低重要性适配器以减少不必要的资源占用，同时保持性能。实验表明，SAFE在保持或优于基线性能的前提下，将内存使用、计算量和训练时间分别降低了42.85%、34.59%和11.82%。此外，我们证明SAFE具有正则化效应，使损失曲面更平滑，帮助模型避开尖锐极小值，从而提升泛化能力。
原文摘要 · Abstract (English)
Transformer-based large-scale pre-trained models achieve great success. Fine-tuning is the standard practice for leveraging these models in downstream tasks. Among the fine-tuning methods, adapter-tuning provides a parameter-efficient fine-tuning by introducing lightweight trainable modules while keeping most pre-trained parameters frozen. However, existing adapter-tuning methods still impose substantial resource usage. Through our investigation, we show that each adapter unequally contributes to both task performance and resource usage. Motivated by this insight, we propose Selective Adapter FrEezing (SAFE), which gradually freezes less important adapters early to reduce unnecessary resource usage while maintaining performance. In our experiments, SAFE reduces memory usage, computation amount, and training time by 42.85\%, 34.59\%, and 11.82\%, respectively, while achieving comparable or better task performance compared to the baseline. We also demonstrate that SAFE induces regularization effect, thereby smoothing the loss landscape, which enables the model to generalize better by avoiding sharp minima.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。