只删敏感词，不伤通用知识，提升大模型删减信息的精准度。
Not Every Token Needs Forgetting: Selective Unlearning to Limit Change in Utility in Large Language Model Unlearning
- 只针对关键敏感词进行遗忘，避免全量更新参数。
- 在两个基准上显著降低对通用知识的损伤，保留模型性能。
- 适合需要精准删除隐私或版权内容的场景使用。
大语言模型（LLM）去学习近年来受到广泛关注，源于需移除模型中的私密、敏感或受版权保护的内容。然而，传统方法无差别地更新所有目标文档中的词元，包括代词、介词等通用词元，这些词元承载着普遍知识。本文指出，并非每个词元都需要遗忘。我们提出选择性去学习（Selective Unlearning, SU），识别与目标信息相关的词元子集，仅对这些词元进行去学习。在两个基准和六种基线去学习算法上的实验表明，SU不仅有效消除目标遗忘数据，还能显著保持模型在保留数据上的性能。
原文摘要 · Abstract (English)
Large Language Model (LLM) unlearning has recently gained significant attention, driven by the need to remove unwanted information, such as private, sensitive, or copyrighted content, from LLMs. However, conventional unlearning approaches indiscriminately update model parameters to forget all tokens in a target document, including common tokens (e.g., pronouns, prepositions, general nouns) that carry general knowledge. In this paper, we highlight that not every token needs forgetting. We propose Selective Unlearning (SU), which identifies a critical subset of tokens within the forgetting set that is relevant to the unwanted information, and unlearns only those tokens. Experiments on two benchmarks and six baseline unlearning algorithms demonstrate that SU not only achieves effective unlearning on the targeted forget data, but also significantly preserves the model's utility in the retaining set.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。