通过平滑注意力分布实现高效遗忘，既删去敏感记忆又保持回答连贯性。
Attention Smoothing Is All You Need For Unlearning
- 利用模型自身注意力构建遗忘教师，通过升高softmax温度平滑分布
- 在多个数据集上实现强遗忘效果，对遗忘提示仍保持回答连贯性
- 适合需要安全可控的LLM应用场景，如隐私保护与内容合规
大型语言模型容易记忆敏感、受版权保护或有害内容，带来重大隐私和法律风险。重新训练成本过高，现有遗忘方法在遗忘与模型性能间权衡不稳定，常导致遗忘提示下输出不连贯，且因注意力中词汇级与语义级关联残留而无法泛化。本文提出注意力平滑遗忘（ASU），将遗忘建模为从模型自身注意力导出的遗忘教师进行自蒸馏。通过提高softmax温度，ASU使注意力分布更平坦，直接抑制重建记忆知识所依赖的词汇级与语义级关联。该方法具备有界优化目标，能有效擦除事实信息，同时维持遗忘提示下的回答连贯性。在TOFU、MUSE和WMDP数据集上的实证评估，以及问答与文本补全任务中的真实场景与持续遗忘测试均表明，ASU在多数场景下优于基线方法，实现鲁棒遗忘且模型效用损失最小。
原文摘要 · Abstract (English)
Large Language Models are prone to memorizing sensitive, copyrighted, or hazardous content, posing significant privacy and legal concerns. Retraining from scratch is computationally infeasible, whereas current unlearning methods exhibit unstable trade-offs between forgetting and utility, frequently producing incoherent outputs on forget prompts and failing to generalize due to the persistence of lexical-level and semantic-level associations in attention. We propose Attention Smoothing Unlearning (ASU), a principled framework that casts unlearning as self-distillation from a forget-teacher derived from the model's own attention. By increasing the softmax temperature, ASU flattens attention distributions and directly suppresses the lexical-level and semantic-level associations responsible for reconstructing memorized knowledge. This results in a bounded optimization objective that erases factual information yet maintains coherence in responses to forget prompts. Empirical evaluation on TOFU, MUSE, and WMDP, along with real-world and continual unlearning scenarios across question answering and text completion, demonstrates that ASU outperforms the baselines for most unlearning scenarios, delivering robust unlearning with minimal loss of model utility.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。