用低秩适配让大模型删知识后仍能高效量化，不丢遗忘效果。
Quantization-Robust LLM Unlearning via Low-Rank Adaptation
- 冻结主模型，只训练可压缩的低秩适配器，确保删知识后参数不被量化抹去。
- 4比特量化下，关键指标最高提升7.93分，隐私泄露降低近20点。
- 适合需要量化部署且要求彻底删知识的场景，如隐私保护应用。
大语言模型遗忘旨在移除训练好的模型中的特定知识，但实际部署常需后训练量化（PTQ）以实现高效推理。然而，激进的低比特PTQ会掩盖遗忘更新，导致量化模型恢复到遗忘前的行为。我们发现，标准全参数微调通常引发的参数变化过小，无法通过4比特量化保留。为此提出基于低秩适配（LoRA）的量化鲁棒遗忘方法：冻结基础模型，将遗忘更新集中于可训练适配器，使有效更新在量化后仍能保持。在Llama-2-7B上使用MUSE数据集（BOOKS和NEWS）评估，LoRA使4比特性能最高提升7.93分（NPO+GDR on BOOKS: 50.17 → 58.10），NEWS上GA+GDR提升4.76分（40.06 → 44.82）。同时，LoRA显著减少4比特下的隐私泄露，例如GA+KLR on BOOKS中，隐私泄露从-25.68降至-5.86（趋近理想0值），且遗忘效果强（VerMem与KnowMem接近0）。因此，在必须量化部署的场景下，使用LoRA进行机器遗忘更具优势。
原文摘要 · Abstract (English)
Large Language Model (LLM) unlearning aims to remove targeted knowledge from a trained model, but practical deployments often require post-training quantization (PTQ) for efficient inference. However, aggressive low-bit PTQ can mask unlearning updates, causing quantized models to revert to pre-unlearning behavior. We show that standard full-parameter fine-tuning often induces parameter changes that are too small to survive 4-bit quantization. We propose quantization-robust unlearning via low-rank adaptation (LoRA): we freeze the base model and concentrate unlearning into trainable adapters so that the effective update is preserved after quantization. On Llama-2-7B evaluated with MUSE dataset (BOOKS and NEWS), LoRA improves 4-bit utility by up to 7.93 points (NPO+GDR on BOOKS: 50.17 to 58.10) and yields higher 4-bit utility on NEWS for GA+GDR (40.06 to 44.82, increase of 4.76). LoRA also substantially reduces privacy leakage under 4-bit PTQ, e.g., for GA+KLR on BOOKS, PrivLeak moves from -25.68 to -5.86 (closer to ideal 0), while maintaining strong forgetting (VerMem and KnowMem near 0). Thus, using LoRA for Machine Unlearning is beneficial for scenarios where quantization is necessary for model deployment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。