用随机初始化的稀疏适配器缓解大模型微调时的遗忘问题。
Random Initialization of Gated Sparse Adapters
- 从随机初始化的全秩适配器出发，通过重零门控和迭代剪枝实现稀疏化。
- 在文本数字识别任务上，比4位QLoRA遗忘更少，尤其在GSM8k上表现更好。
- 适合关注参数效率与记忆保持平衡的研究者或应用开发者。
在新任务上微调语言模型时，灾难性遗忘——即对先前任务性能下降——是普遍存在的问题。尽管参数高效微调（PEFT）方法如LoRA通过低秩适配器缓解该问题，稀疏适配则提供了一种不施加秩约束的替代方案。我们提出随机初始化的门控稀疏适配器（RIGSA），从随机初始化的全秩适配器开始，使用类似ReZero的门控机制，并通过迭代幅度剪枝实现稀疏化。我们在SmolLM2-1.7B-Instruct上评估RIGSA，采用一种新颖的图文任务（Textual MNIST），并在PIQA、HellaSwag和GSM8k上测量遗忘情况。SmolLM2-1.7B-Instruct在初始时对Textual MNIST表现接近随机水平，但可通过RIGSA、4-bit QLoRA和随机掩码学习该任务。尽管相比QLoRA拥有更多可训练参数，我们研究的RIGSA配置在遗忘方面优于QLoRA，尤其是在GSM8k上，与随机掩码表现相当。
原文摘要 · Abstract (English)
When fine-tuning language models on new tasks, catastrophic forgetting -- performance degradation on previously-learned tasks -- is a ubiquitous problem. While Parameter-Efficient Fine-Tuning (PEFT) methods like LoRA address this through low-rank adapters, sparse adaptation offers an alternative that doesn't impose rank constraints. We introduce Random Initialization of Gated Sparse Adapters (RIGSA), which starts from randomly-initialized full-rank adapters, gates them with a ReZero analog, and sparsifies them with iterative magnitude pruning. We evaluate RIGSA on SmolLM2-1.7B-Instruct using a novel vision-in-text task (Textual MNIST) and measure forgetting on PIQA, HellaSwag, and GSM8k. SmolLM2-1.7B-Instruct initially performs around chance level on Textual MNIST, and is capable of learning the task through RIGSA, 4-bit QLoRA and random masking. In spite of having more trainable parameters than QLoRA, the RIGSA configurations that we studied displayed less forgetting than QLoRA, particularly on GSM8k, though it performs comparably to random masking.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。