让大模型删除特定数据，同时保持能力不下降。
OBLIVIATE: Robust and Practical Machine Unlearning for Large Language Models
- 用提取关键词+保留集+三组件损失函数实现精准删忆
- 在哈利·波特等数据上删忆效果显著，且不影响其他能力
- 采用低秩适配器，高效实用适合工业场景
大规模语言模型在训练过程中可能记忆敏感、版权或有害内容。为此，我们提出OBLIVIATE框架，可在不损害模型性能的前提下，有效移除特定数据。该框架包含三个步骤：提取目标词元、构建保留数据集，并使用包含掩码、知识蒸馏和世界事实三个成分的定制损失函数进行微调。通过引入低秩适配器（LoRA）保证效率，同时维持删忆质量。我们在哈利·波特系列、WMDP和TOFU等多个数据集上进行实验，采用文档级记忆评分、模型效用和流畅性等综合指标评估。结果表明，该方法能有效抵御成员推理攻击，对保留数据影响小，在多种场景下表现稳健。
原文摘要 · Abstract (English)
Large language models (LLMs) trained over extensive corpora risk memorizing sensitive, copyrighted, or toxic content. To address this, we propose \textbf{OBLIVIATE}, a robust unlearning framework that removes targeted data while preserving model utility. The framework follows a structured process: extracting target tokens, building retain sets, and fine-tuning with a tailored loss function comprising three components -- masking, distillation, and world fact. Using low-rank adapters (LoRA) ensures efficiency without compromising unlearning quality. We conduct experiments on multiple datasets, including Harry Potter series, WMDP, and TOFU, using a comprehensive suite of metrics: \emph{forget quality} (via a new document-level memorization score), \emph{model utility}, and \emph{fluency}. Results demonstrate its effectiveness in resisting membership inference attacks, minimizing the impact on retained data, and maintaining robustness across diverse scenarios.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。