训练到模型'顿悟'后进行机器遗忘，效果更好更稳定。
Grokked Models are Better Unlearners
- 在模型完成'顿悟'后才开始遗忘，可实现更精准的删除。
- 遗忘所需更新次数减少，且对其他数据影响更小。
- 适合希望高效安全删除数据的研究者与工程师。
模型在训练后期出现的'顿悟'现象（grokking）——即在完全拟合训练数据后仍持续提升泛化能力——被认为与鲁棒性和表征质量有关。本文探讨该训练阶段是否有助于机器未学习（machine unlearning），即在不重新训练的情况下移除特定数据的影响。我们在视觉任务（CNN/ResNet 在 CIFAR、SVHN、ImageNet）和语言任务（基于 TOFU 风格的 Transformer）上对比了在'顿悟'前与'顿悟'后应用标准未学习方法的效果。结果表明，从已'顿悟'的检查点开始，能（i）以更少的更新达到目标遗忘水平，（ii）对保留数据和测试性能的负面影响更小，（iii）在不同随机种子下更新更稳定。特征与曲率分析显示，'顿悟'后模型学习到更模块化的表示，且遗忘与保留子集间的梯度对齐程度降低，从而支持选择性遗忘。研究揭示了训练时机（预或后顿悟）是优化未学习策略的重要独立因素，为改进现有方法提供了无需改动算法的实际方案。
原文摘要 · Abstract (English)
Grokking-delayed generalization that emerges well after a model has fit the training data-has been linked to robustness and representation quality. We ask whether this training regime also helps with machine unlearning, i.e., removing the influence of specified data without full retraining. We compare applying standard unlearning methods before versus after the grokking transition across vision (CNNs/ResNets on CIFAR, SVHN, and ImageNet) and language (a transformer on a TOFU-style setup). Starting from grokked checkpoints consistently yields (i) more efficient forgetting (fewer updates to reach a target forget level), (ii) less collateral damage (smaller drops on retained and test performance), and (iii) more stable updates across seeds, relative to early-stopped counterparts under identical unlearning algorithms. Analyses of features and curvature further suggest that post-grokking models learn more modular representations with reduced gradient alignment between forget and retain subsets, which facilitates selective forgetting. Our results highlight when a model is trained (pre- vs. post-grokking) as an orthogonal lever to how unlearning is performed, providing a practical recipe to improve existing unlearning methods without altering their algorithms.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。