提出更稳定有效的遗忘方法，能持久删除信息且不损害模型能力。
Unlearning That Lasts: Utility-Preserving, Robust, and Almost Irreversible Forgetting in LLMs
- 用杰恩斯-申农散度优化遗忘与保留数据，提升训练稳定性。
- 在真实场景下验证，遗忘效果优于现有方法且抵抗意外重学。
- 构建新评估体系，用大模型判别+对抗性测试，发现旧方法虚高。
大型语言模型（LLMs）中的遗忘技术旨在精确移除预训练模型中的特定信息，这对保障模型安全、删除私密数据或有害知识至关重要。然而，现有遗忘方法在严格评估下常表现不佳。为此，我们提出JensUn，利用杰恩斯-申农散度（Jensen-Shannon Divergence）作为遗忘集和保留集的训练目标，相比常用损失函数，实现更稳定高效的遗忘动态。在广泛实验中，JensUn在遗忘-性能权衡上优于对比方法，并展现出对良性重学的强韧性。此外，为实现精准遗忘评估，我们构建了LKF——一个包含较少为人知事实的精选数据集，提供更真实的遗忘场景。最后，为全面检验遗忘方法，我们提出：(i) 使用大模型作为语义判别器替代传统ROUGE分数；(ii) 在多种改写和输入格式下进行最坏情况下的遗忘评估。新评估框架揭示，许多现有方法的实际效果远低于此前认知。
原文摘要 · Abstract (English)
Unlearning in large language models (LLMs) involves precisely removing specific information from a pre-trained model. This is crucial to ensure safety of LLMs by deleting private data or harmful knowledge acquired during pre-training. However, existing unlearning methods often fall short when subjected to thorough evaluation. To overcome this, we introduce JensUn, where we leverage the Jensen-Shannon Divergence as the training objective for both forget and retain sets for more stable and effective unlearning dynamics compared to commonly used loss functions. In extensive experiments, JensUn achieves better forget-utility trade-off than competing methods, and even demonstrates strong resilience to benign relearning. Additionally, for a precise unlearning evaluation, we introduce LKF, a curated dataset of lesser-known facts that provides a realistic unlearning scenario. Finally, to comprehensively test unlearning methods, we propose (i) employing an LLM as semantic judge instead of the standard ROUGE score, and (ii) using worst-case unlearning evaluation over various paraphrases and input formats. Our improved evaluation framework reveals that many existing methods are less effective than previously thought.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。