无需保留集即可高效删除大模型记忆,避免性能下降。
SHRED: Retain-Set-Free Unlearning via Self-Distillation with Logit Demotion

- 通过高困惑度词定位需遗忘内容,低困惑度词作为安全锚点。
- 单次自蒸馏训练实现遗忘与性能保持的平衡,优于依赖保留集的方法。
- 适合需要安全删除数据且无额外数据可用的场景。
大语言模型的机器遗忘旨在选择性移除私密数据、版权文本或危险知识,而无需昂贵的全量重训练。现有方法多依赖精心筛选的保留集以防止通用能力崩溃,带来额外的数据依赖。本文提出SHRED(基于高困惑度仅保留集免去熵降低的自蒸馏),核心洞察是遗忘样本中并非所有标记都携带同等记忆信息:高信息标记集中体现模型记忆,低信息标记反映通用语言能力。SHRED分两阶段运行:(1) 选择阶段,在遗忘样本上进行前向传播,收集逐标记自回归概率,选取概率最低(香农信息最高)的标记为遗忘位置,其余位置保留为良性锚点;(2) 训练阶段,构建修正后的KL目标,在遗忘位置降低记忆标记的对数几率,同时在良性位置保持原始分布。模型通过单一顶层KL自蒸馏目标训练,同步实现遗忘与效用保持。我们在四个标准遗忘基准上评估,结果表明其在遗忘效果与模型效用间达成新的帕累托最优,优于依赖保留集的方法。分析显示,SHRED对重学习攻击和成员推断攻击具有鲁棒性,且在多次连续遗忘后仍保持稳定性能。
原文摘要 · Abstract (English)
Machine unlearning for large language models (LLMs) aims to selectively remove memorized content such as private data, copyrighted text, or hazardous knowledge, without costly full retraining. Most existing methods require a retain set of curated examples to prevent catastrophic degradation of general model utility, creating an extra data dependency that complicates deployment. We propose SHRED (Self-distillation via High-surprisal-only Retain-set-free Entropy Demotion), a retain-set-free unlearning method built on a key insight: not all tokens within a forget set instance carry memorized information equally. High-information tokens concentrate the model's memorized knowledge, while low-information tokens reflect general language competence. SHRED operates in two stages. (1) Selection: We perform a forward pass on a forget set instance, collect per-token autoregressive probabilities, and select the bottom (lowest probability, highest Shannon information) as forget positions; the remaining positions are retained as benign anchors. (2) Training: We construct modified KL targets that demote the memorized token's logit at forget positions while preserving the original distribution at benign positions. The model is then trained via a single top KL self-distillation objective that simultaneously drives forgetting and utility preservation. We evaluate SHRED across four standard unlearning benchmarks and demonstrate that it establishes a new Pareto-optimal trade-off between forget efficacy and model utility, outperforming retain-set-dependent methods. Our analysis shows that SHRED is robust against relearning attacks and membership-inference attacks, and it maintains stable utility even after many sequential unlearning runs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。