让大模型精准遗忘敏感信息，同时保留正常语言能力。
Unlearning Is Not Just Erasing: Temporal Decoupling via Generation Inequality
- 通过解耦注意力路径实现细粒度遗忘，而非简单删除内容。
- 在TOFU基准上遗忘质量达0.93，保持87%~98%的模型实用性。
- 适合需要隐私保护且不希望损伤模型整体能力的研究者。
大型语言模型需有效实现遗忘以应对隐私法规与安全问题。现有序列和标记级方法仅惩罚目标输出，未建模其上下文依赖的检索路径，易破坏语言结构或抑制良性知识。我们提出ADU，一种基于训练的细粒度框架，将遗忘从标记擦除转变为上下文注意力路径解耦。利用局部与全局注意力头的功能差异，ADU识别出持续敏感锚点的预选位置，并固定其原始模型下的候选路径。随后训练注意力投影适配器，抑制这些路径上的注意力质量，同时保留局部注意力结构和保留集的语言建模能力。后训练激活交换测试验证修改后的注意力输出模块是否传递了学习到的遗忘效果。ADU在TOFU和WMDP基准上表现最优，于TOFU上达到0.93的遗忘质量；在保持87%~98%模型实用性的同时（平均92.9%，基线81.9%），显著减少良性上下文中的副作用。
原文摘要 · Abstract (English)
Large language models (LLMs) require effective unlearning to address privacy regulations and safety concerns. However, achieving precise forgetting without compromising general utility remains challenging. Existing sequence- and token-level methods penalize target outputs without modeling their context-dependent retrieval paths, which can disrupt linguistic structure or suppress benign knowledge. We present ADU, a fine-grained, training-based framework that shifts unlearning from token erasure to contextual attention-pathway decoupling. Exploiting the functional distinction between local and global attention heads, ADU identifies preplan positions that retrieve persistent sensitive anchors and fixes their candidate paths under the original model. It then trains attention-projection adapters to suppress attention mass along these paths while preserving local-attention structure and retain-set language modeling. Post-training activation exchange tests whether the modified attention-output module transmits the learned forgetting effect. ADU achieves the strongest aggregate performance among evaluated baselines on the TOFU and WMDP benchmarks, including a Forget Quality of (0.93) on TOFU. It preserves 87--98% of model utility (92.9% on average versus 81.9% for baselines) while reducing side effects in benign contexts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。