用随机门控稀疏化冻结模型，小参数实现高效微调。
FineGates: LLMs Finetuning with Compression using Stochastic Gates
- 通过随机门控机制动态激活基模型部分参数，实现稀疏化微调。
- 在保持竞争力准确率的同时，可移除20%-40%模型参数。
- 适合资源受限场景，尤其适用于小数据集下的轻量级微调。
大型语言模型（LLMs）因参数量巨大，在全量微调时面临高计算开销、内存需求和实际应用不可行的问题。在计算资源有限或数据量较小时，更新全部参数易导致过拟合。为此，已有轻量微调方法如低秩适配器层被提出，仅训练少量附加参数并冻结基础模型，以降低资源消耗并缓解过拟合风险。本文提出一种基于随机门控的适配器模型，可在任务特定适配的同时对冻结的基础模型进行稀疏化。该方法仅需少量可训练参数，且能加速基础模型推理，同时保持较高准确率。我们在多种变体中引入额外低秩参数，并与多个近期基线方法对比。结果表明，所提方法在微调模型准确率上优于多个基线，且可移除20%-40%的模型参数而无显著精度损失。
原文摘要 · Abstract (English)
Large Language Models (LLMs), with billions of parameters, present significant challenges for full finetuning due to the high computational demands, memory requirements, and impracticality of many real-world applications. When faced with limited computational resources or small datasets, updating all model parameters can often result in overfitting. To address this, lightweight finetuning techniques have been proposed, like learning low-rank adapter layers. These methods aim to train only a few additional parameters combined with the base model, which remains frozen, reducing resource usage and mitigating overfitting risks. In this work, we propose an adaptor model based on stochastic gates that simultaneously sparsify the frozen base model with task-specific adaptation. Our method comes with a small number of trainable parameters and allows us to speed up the base model inference with competitive accuracy. We evaluate it in additional variants by equipping it with additional low-rank parameters and comparing it to several recent baselines. Our results show that the proposed method improves the finetuned model accuracy comparatively to the several baselines and allows the removal of up to 20-40\% without significant accuracy loss.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。