提出简单高效的稀疏微调方法，显著提升大模型训练效率
Refining Salience-Aware Sparse Fine-Tuning Strategies for Language Models
- 用梯度信号指导稀疏参数选择，静态掩码保持性能同时加速训练
- 在多个NLP任务上优于传统微调方法，精度与效率双优
- 开源框架支持复现，为高效微调提供新基准
参数高效微调（PEFT）因低秩适配（LoRA）等方法而备受关注。本文聚焦基于稀疏性的PEFT（SPEFT），通过在模型权重矩阵中引入可训练稀疏适配，相比低秩方法更具参数选择灵活性。我们首次系统评估了用于SPEFT的显著性度量，受零成本NAS代理启发，发现简单的基于梯度的度量表现可靠，效果与最佳替代方案相当，兼具计算效率与鲁棒性能。此外，对比静态与动态掩码策略，发现静态掩码（训练前预设非零参数）在不损失性能的前提下实现更高效率，而动态掩码无明显优势。在多个NLP任务中，一种基于梯度的静态SPEFT方法持续优于其他微调方法，为大模型提供了简单但有效的基准。本工作挑战了‘复杂即有效’的固有观念，开源框架[https://github.com/0-ml/speft]建立了可复现的基准，推动未来研究。
原文摘要 · Abstract (English)
Parameter-Efficient Fine-Tuning (PEFT) has gained prominence through low-rank adaptation methods like LoRA. In this paper, we focus on sparsity-based PEFT (SPEFT), which introduces trainable sparse adaptations to the weight matrices in the model, offering greater flexibility in selecting fine-tuned parameters compared to low-rank methods. We conduct the first systematic evaluation of salience metrics for SPEFT, inspired by zero-cost NAS proxies, and identify simple gradient-based metrics is reliable, and results are on par with the best alternatives, offering both computational efficiency and robust performance. Additionally, we compare static and dynamic masking strategies, finding that static masking, which predetermines non-zero entries before training, delivers efficiency without sacrificing performance, while dynamic masking offers no substantial benefits. Across NLP tasks, a simple gradient-based, static SPEFT consistently outperforms other fine-tuning methods for LLMs, providing a simple yet effective baseline for SPEFT. Our work challenges the notion that complexity is necessary for effective PEFT, while our open-source framework establishes a reproducible benchmark for future research, which is available at [https://github.com/0-ml/speft].
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。