用低秩更新技术实现高效删模，既省资源又保性能。
LoRA Unlearns More and Retains More (Student Abstract)
- 先剪枝再适配后删模，三步流程降低计算开销。
- 在多个数据集上优于现有近似删模方法，接近精确删模效果。
- 适合需要合规删模且注重推理性能的工业场景。
随着隐私法规日益严格，机器删模（MU）变得至关重要。其目标是移除模型中与特定类别相关的信息。传统方法通过在剩余数据集上重新训练实现精确删模，但计算成本高昂。这推动了更高效的删模技术发展，包括模型稀疏化方法，但会损害模型在剩余类别上的性能。为此，我们提出一种新方法 PruneLoRA，引入一种称为“先剪枝、再适配、后删模”的新型删模范式。LoRA（Hu 等，2022）通过低秩更新减少大规模参数调整需求。我们利用 LoRA 对剪枝后的模型参数进行选择性修改，从而降低计算成本和内存占用，并提升模型在剩余类别上的性能保留能力。实验结果表明，该方法在多种指标上优于其他近似删模方法，缩小了近似与精确删模之间的差距。代码已开源：https://github.com/vlgiitr/LoRA-Unlearn。
原文摘要 · Abstract (English)
Due to increasing privacy regulations and regulatory compliance, Machine Unlearning (MU) has become essential. The goal of unlearning is to remove information related to a specific class from a model. Traditional approaches achieve exact unlearning by retraining the model on the remaining dataset, but incur high computational costs. This has driven the development of more efficient unlearning techniques, including model sparsification techniques, which boost computational efficiency, but degrade the model's performance on the remaining classes. To mitigate these issues, we propose a novel method, PruneLoRA which introduces a new MU paradigm, termed prune first, then adapt, then unlearn. LoRA (Hu et al. 2022) reduces the need for large-scale parameter updates by applying low-rank updates to the model. We leverage LoRA to selectively modify a subset of the pruned model's parameters, thereby reducing the computational cost, memory requirements and improving the model's ability to retain performance on the remaining classes. Experimental Results across various metrics showcase that our method outperforms other approximate MU methods and bridges the gap between exact and approximate unlearning. Our code is available at https://github.com/vlgiitr/LoRA-Unlearn.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。