让稀疏大模型也能高效删去敏感信息，不重训也能忘得干净。
SAU: Sparsity-Aware Unlearning for LLMs via Gradient Masking and Importance Redistribution
- 用梯度掩码把更新引导到存活参数上，避开被剪枝的权重。
- 在稀疏模型上实现接近全量模型的遗忘效果，同时保持性能稳定。
- 适合需要隐私保护且部署时启用稀疏化的大型语言模型用户。
大语言模型在训练过程中不可避免会记忆敏感信息，带来严重隐私风险。机器遗忘作为无需全量重训即可选择性清除这些信息的解决方案日益受到关注。然而，现有方法针对稠密模型设计，忽视了模型稀疏化这一高效部署的关键技术。我们发现，在稀疏模型上，现有遗忘方法的有效性显著下降。实证分析表明，这是因为现有方法需更新所有参数，而稀疏化将大量权重置零，从根本上限制了模型的遗忘能力。为此，我们提出稀疏感知遗忘（SAU），通过梯度掩码将更新导向存活权重，并结合重要性感知重分配补偿被剪枝参数的影响，实现遗忘与稀疏目标的解耦。大量实验表明，SAU在稀疏LLM上显著优于现有方法，既能有效遗忘，又保持模型可用性。
原文摘要 · Abstract (English)
Large Language Models (LLMs) inevitably memorize sensitive information during training, posing significant privacy risks. Machine unlearning has emerged as a promising solution to selectively remove such information without full retraining. However, existing methods are designed for dense models and overlook model sparsification, an essential technique for efficient LLM deployment. We find that unlearning effectiveness degrades substantially on sparse models. Through empirical analysis, we reveal that this degradation occurs because existing unlearning methods require updating all parameters, yet sparsification prunes substantial weights to zero, fundamentally limiting the model's forgetting capacity. To address this challenge, we propose Sparsity-Aware Unlearning (SAU), which decouples unlearning from sparsification objectives through gradient masking that redirects updates to surviving weights, combined with importance-aware redistribution to compensate for pruned parameters. Extensive experiments demonstrate that SAU significantly outperforms existing methods on sparse LLMs, achieving effective forgetting while preserving model utility.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。