提出新剪枝方法，有效缓解大模型微调时的灾难性遗忘。
Mitigating Catastrophic Forgetting in Large Language Models with Forgetting-aware Pruning
- 基于任务向量与预训练参数重叠度设计剪枝指标
- 在8个数据集上遗忘率仅0.25%，下游任务准确率达99.67%
- 无需修改训练流程或额外数据，适合持续学习场景
大型语言模型在微调过程中常面临灾难性遗忘问题。本文提出遗忘感知剪枝度量（FAPM），通过任务向量与预训练参数的重叠程度量化遗忘风险，并将其融入剪枝标准。该方法无需修改训练过程、模型结构或引入辅助数据。在涵盖自然语言推理、通用问答、医学问答、数学问答、阅读理解及完形填空等8个数据集上的实验表明，FAPM将遗忘率控制在0.25%以下，同时保持下游任务99.67%的准确率。代码已公开。
原文摘要 · Abstract (English)
Recent advancements in large language models (LLMs) have shown impressive capabilities in various downstream tasks but typically face Catastrophic Forgetting (CF) during fine-tuning. In this paper, we propose the Forgetting-Aware Pruning Metric (FAPM), a novel pruning-based approach to balance CF and downstream task performance. Our investigation reveals that the degree to which task vectors (i.e., the subtraction of pre-trained weights from the weights fine-tuned on downstream tasks) overlap with pre-trained model parameters is a critical factor for CF. Based on this finding, FAPM employs the ratio of the task vector to pre-trained model parameters as a metric to quantify CF, integrating this measure into the pruning criteria. Importantly, FAPM does not necessitate modifications to the training process or model architecture, nor does it require any auxiliary data. We conducted extensive experiments across eight datasets, covering natural language inference, General Q&A, Medical Q&A, Math Q&A, reading comprehension, and cloze tests. The results demonstrate that FAPM limits CF to just 0.25\% while maintaining 99.67\% accuracy on downstream tasks. We provide the code to reproduce our results.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。