arXiv:2601.21682cs.CLcs.AI2026-01

提出新框架,让大模型持续删除敏感内容而不遗忘有用知识。

FIT to Forget: Robust Continual Unlearning for Large Language Models

  • 用过滤、重要性选择和层定位三机制稳定连续删改过程。
  • 处理数百次请求后仍保持高下游任务性能(如GSM8K)。
  • 适合需要长期安全合规的大模型应用者使用。

大型语言模型虽能力强大,但日益面临删除记忆中的隐私、版权或有害内容的需求。现有方法多针对单次删除,而真实场景中删除请求是持续到来的。若直接套用旧方法,会导致性能严重下降和灾难性遗忘。为此,本文提出 it框架,可有效处理高密度连续删除流,同时抵抗灾难性遗忘与事后恢复。该框架通过冗余过滤、重要性感知自适应算法选择及目标层归因三个协同机制实现更新稳定。为支持严格评估,我们构建了统一基准PCH,涵盖个人、版权与有害内容,并采用对称指标忘却度(F.D.)与保留效用(R.U.)系统量化遗忘-效用权衡。在五个大模型(最大140亿参数)上实验证明, it在持续删除下均达到当前最优的去记忆效果与性能保持。即使经历数百次连续请求,仍能维持强下游表现(如GSM8K、MMLU),且对重学与量化恢复攻击具有更强鲁棒性。

原文摘要 · Abstract (English)

While large language models (LLMs) exhibit remarkable capabilities, they increasingly face demands to unlearn memorized privacy-sensitive, copyrighted, or harmful content. Existing unlearning methods primarily focus on \emph{single-shot} scenarios, whereas real-world deletion requests arrive \emph{continually}. Naïvely applying these methods to sequential requests leads to severe utility degradation and catastrophic forgetting. To address this, we propose \fit, a robust continual unlearning framework to process high-volume sequential deletion streams while resisting both catastrophic forgetting and post-unlearning recovery. \fit stabilizes sequential updates through three synergistic mechanisms: redundancy \underline{F}iltering, \underline{I}mportance-aware adaptive algorithm selection, and \underline{T}argeted layer attribution. Furthermore, to facilitate rigorous evaluation, we introduce \textbf{PCH}, a unified benchmark encompassing \textbf{P}ersonal, \textbf{C}opyrighted, and \textbf{H}armful content, alongside two symmetric metrics, Forget Degree (F.D.) and Retain Utility (R.U.), to systematically quantify forgetting-utility trade-offs. Extensive experiments across five LLMs (up to 14B parameters) demonstrate that \fit consistently achieves state-of-the-art unlearning efficacy and utility preservation. Notably, even after hundreds of sequential requests, \fit preserves strong downstream (\eg, GSM8K, MMLU) performance and exhibits superior resilience against relearning and quantization recovery attacks.

大模型持续学习去记忆

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。