ReLearn通过数据增强与微调实现大模型精准删忆，兼顾遗忘效果与生成质量。
ReLearn: Unlearning via Learning for Large Language Models
- 用数据增强与微调替代反向优化，避免破坏文本连贯性。
- 在目标知识遗忘率超90%时仍保持生成流畅度，优于现有方法。
- 提出KFR/KRR和LS指标，更全面评估遗忘与生成能力。
当前大语言模型的遗忘方法多依赖反向优化降低目标词概率，但会破坏后续词汇预测，导致模型性能下降和语言连贯性受损。此外，现有评估指标过度关注上下文遗忘，却忽视生成流畅性与相关性。为此，我们提出ReLearn，一种基于数据增强与微调的高效遗忘流水线，并构建综合评估框架。该框架引入知识遗忘率（KFR）与知识保留率（KRR）衡量知识层面的保持情况，以及语言评分（LS）评估生成质量。实验表明，ReLearn在实现目标遗忘的同时，仍能保持高质量输出。机制分析进一步揭示反向优化如何破坏连贯生成，而ReLearn有效保留此关键能力。代码已开源：https://github.com/zjunlp/unlearn。
原文摘要 · Abstract (English)
Current unlearning methods for large language models usually rely on reverse optimization to reduce target token probabilities. However, this paradigm disrupts the subsequent tokens prediction, degrading model performance and linguistic coherence. Moreover, existing evaluation metrics overemphasize contextual forgetting while inadequately assessing response fluency and relevance. To address these challenges, we propose ReLearn, a data augmentation and fine-tuning pipeline for effective unlearning, along with a comprehensive evaluation framework. This framework introduces Knowledge Forgetting Rate (KFR) and Knowledge Retention Rate (KRR) to measure knowledge-level preservation, and Linguistic Score (LS) to evaluate generation quality. Our experiments show that ReLearn successfully achieves targeted forgetting while preserving high-quality output. Through mechanistic analysis, we further demonstrate how reverse optimization disrupts coherent text generation, while ReLearn preserves this essential capability. Code is available at https://github.com/zjunlp/unlearn.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。