让稀疏大模型在微调中动态调整结构，兼顾性能与效率
Leave it to the Specialist: Repair Sparse LLMs with Sparse Fine-Tuning via Sparsity Evolution
- 通过稀疏演化机制动态重分配任务更新并激活被剪枝权重
- 在多个模型和基准上实现更强性能，内存与时间效率更优
- 适合追求高效部署的稀疏大模型研究者与应用开发者
稀疏大语言模型（LLMs）为高效部署提供了可行方向，但将其适配到下游任务仍具挑战。核心难点在于如何在不牺牲稀疏性优势的前提下实现有效任务适应。现有微调方法不适用：要么引入额外密集参数，要么假设稀疏结构固定，难以兼容稀疏LLM。本文提出专为稀疏LLM设计的稀疏演化微调（SEFT）框架。SEFT在微调过程中允许稀疏结构动态演化，通过周期性重新分配任务特定更新，并在有益时重新激活先前剪枝的权重。同时，基于参数重要性进行拓扑自适应，保持稀疏性带来的效率优势。在LLaMA、DeepSeek和Mistral模型上，多个基准测试表明，SEFT在性能上优于现有基线，且具备更优的内存与时间效率。代码已公开于：https://github.com/QiaoXiao7282/SEFT。
原文摘要 · Abstract (English)
Sparse large language models (LLMs) offer an attractive direction toward efficient deployment, but adapting them to downstream tasks remains challenging. The central difficulty is to enable effective task adaptation without sacrificing the efficiency advantages of sparsity. Existing fine-tuning methods are not well-suited to this setting, as they either introduce additional dense parameters or assume a fixed sparse topology, limiting their compatibility with sparse LLMs. In this paper, we propose Sparsity Evolution Fine-Tuning (SEFT), a fine-tuning framework designed specifically for sparse LLMs. SEFT allows sparse structure to evolve during fine-tuning by periodically reallocating sparse task-specific updates and reactivating previously pruned weights when beneficial. At the same time, SEFT preserves the efficiency advantages of sparsity through topology adaptation based on parameter importance. Experiments on LLaMA, DeepSeek, and Mistral models across multiple benchmarks show that SEFT delivers stronger performance while offering superior memory and time efficiency compared to existing baselines. Our code is publicly available at: https://github.com/QiaoXiao7282/SEFT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。