提出新指标预测大模型遗忘后是否会被轻易恢复知识。
Distance Is Not Enough: Forget-Retain Alignment Gap Predicts LLM Relearning Robustness
- 用遗忘-保留权重对齐度评估更新效果，比全局距离更可靠。
- 该方法无需重训练即可预测模型是否易被复学知识。
- 适合研究模型可解释性与安全遗忘的开发者参考。
机器遗忘旨在让模型删除特定数据，但已遗忘的大模型常会因微调而重新获得知识。现有鲁棒性预测依赖全局权重空间位移，但距离本身可能误导——随机或破坏性更新会导致性能下降。我们提出，再学习鲁棒性取决于更新结构：真正鲁棒的遗忘应只影响关键遗忘权重，同时保护关键保留权重。为此，我们引入无需训练的预测器FRAG（Forget-Retain Alignment Gap），能准确区分选择性更新与密集更新，优于全局距离。基于遗忘关键、保留安全的原则，我们提出遗忘-保留剪枝（FRP）方法，显著提升再学习鲁棒性。结果表明，权重选择性比距离更能解释鲁棒性。代码已开源：https://github.com/Yi1-Chen/FRAG。
原文摘要 · Abstract (English)
Machine unlearning aims to make a model forget specific data, yet unlearned LLMs often fail to stay unlearned: brief fine-tuning can revive removed knowledge. Existing robustness predictors rely on global weight-space displacement, but distance alone can be misleading when random or destructive updates collapse performance. We argue that relearning robustness depends on update structure: robust unlearning should affect forget-critical weights while sparing retain-critical ones. We introduce the Forget-Retain Alignment Gap (FRAG), a training-free predictor that scores an update's forget-retain alignment without running a relearning attack, and separates selective from dense updates more reliably than global distance. Building on the forget-critical, retain-sparing principle, Forget-Retain Pruning (FRP) improves relearning robustness. Our results suggest that weight selectivity better explains robustness than distance alone. Code is available at https://github.com/Yi1-Chen/FRAG.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。