通过神经网络剪枝识别关键节点，实现更高效的稀疏微调。
An Efficient Sparse Fine-Tuning with Low Quantization Error via Neural Network Pruning
- 基于剪枝方法筛选重要神经元，仅微调相关权重。
- 内存效率提升20%-50%，精度媲美LoRA等先进方法。
- 适合资源有限但需高效微调的场景，如边缘设备部署。
微调是将大语言模型等基础模型适配到下游任务的关键步骤。为让计算资源有限的用户也能使用，开发内存与计算高效的微调方法至关重要。稀疏微调（SpFT）和低秩适应（LoRA）是两种广泛采用的解决方案。本文提出一种新SpFT框架，借鉴神经网络剪枝思想：首先利用剪枝中的特征重要性度量（具体采用结构化剪枝方法）识别关键神经元，随后仅对涉及这些神经元的权重进行微调。在常见语言任务上的实验表明，该方法在保持与最先进方法（如LoRA变体）相当精度的同时，使内存效率提升20%-50%。代码已开源：https://github.com/CenjhihLi/sparsity_finetuning。
原文摘要 · Abstract (English)
Fine-tuning is an important step in adapting foundation models such as large language models to downstream tasks. To make this step more accessible to users with limited computational budgets, it is crucial to develop fine-tuning methods that are memory and computationally efficient. Sparse Fine-tuning (SpFT) and Low-rank adaptation (LoRA) are two frameworks that have emerged for addressing this problem and have been adopted widely in practice. In this work, we develop a new SpFT framework, based on ideas from neural network pruning. At a high level, we first identify "important" neurons/nodes using feature importance metrics from network pruning (specifically, we use the structural pruning method), and then perform fine-tuning by restricting to weights involving these neurons. Experiments on common language tasks show our method improves SpFT's memory efficiency by 20-50\% while matching the accuracy of state-of-the-art methods like LoRA's variants. Code available at: https://github.com/CenjhihLi/sparsity_finetuning
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。