通过稀疏化实现高效微调，少训练多加速。
Train Less, Infer Faster: Efficient Model Finetuning and Compression via Structured Sparsity
- 用随机门控稀疏化模型行与列，几乎不需额外参数训练。
- 减少20%~40%参数量，推理速度提升，准确率损失小。
- 适合资源受限场景，对模型压缩与快速适配有优势。
全量微调千亿参数的预训练语言模型因计算成本高、内存占用大且易过拟合而难以实施。尽管低秩适配器（LoRA）通过添加小型可训练模块缓解了部分问题，但其仍增加内存开销且不降低推理延迟。本文发现：对特定模型行和列进行稀疏化即可实现高效任务适配，无需权重调整。我们提出基于训练时随机门控的稀疏化微调方案，仅需极少可训练参数，减少20%–40%模型参数，显著降低推理时间，且准确率损失轻微。实验证明，该方法在效率与性能上优于近期主流微调基线。此外，我们提供了该随机门控过程的收敛性理论保证，并表明其优化景观更简单、条件更好于LoRA。结果凸显稀疏性在语言模型任务适配中的潜力。
原文摘要 · Abstract (English)
Fully finetuning foundation language models (LMs) with billions of parameters is often impractical due to high computational costs, memory requirements, and the risk of overfitting. Although methods like low-rank adapters help address these challenges by adding small trainable modules to the frozen LM, they also increase memory usage and do not reduce inference latency. We uncover an intriguing phenomenon: sparsifying specific model rows and columns enables efficient task adaptation without requiring weight tuning. We propose a scheme for effective finetuning via sparsification using training stochastic gates, which requires minimal trainable parameters, reduces inference time, and removes 20--40\% of model parameters without significant accuracy loss. Empirical results show it outperforms recent finetuning baselines in efficiency and performance. Additionally, we provide theoretical guarantees for the convergence of this stochastic gating process, and show that our method admits a simpler and better-conditioned optimization landscape compared to LoRA. Our results highlight sparsity as a compelling mechanism for task-specific adaptation in LMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。