提出稳定遗忘方法，解决大模型删数据时参数失控问题。
Stable Forgetting: Bounded Parameter-Efficient Unlearning in Foundation Models
- 用有界函数控制遗忘时的权重更新，防止梯度爆炸。
- 在多个视觉与语言模型上实现高遗忘率且保持模型性能。
- 适合需要安全删数据的大模型应用，如隐私保护场景。
基础模型（如语言和视觉Transformer）中的机器遗忘对隐私与安全至关重要，但现有方法不稳定不可靠。广泛采用的梯度差法在保留数据上做梯度下降，对遗忘数据做梯度上升，当结合交叉熵损失时，会导致权重和梯度无界增长，损害遗忘与保留效果。我们提供理论框架，揭示上升操作如何在Transformer前馈MLP层中破坏优化稳定性。基于此，提出*有界参数高效遗忘*（Bounded Parameter-Efficient Unlearning），通过在LoRA适配器上施加有界函数，控制上升过程中的权重动态，实现可靠收敛。在CIFAR-100上验证，针对ViT-B/16、ViT-L/14和DeiT-S架构，仅GD+Sine方法同时实现高遗忘质量与模型效用；并在语言模型基准TOFU、TDEC、MUSE上展示泛化性，覆盖22M至8B参数模型，实现更优遗忘同时保持性能。
原文摘要 · Abstract (English)
Machine unlearning in foundation models (e.g., language and vision transformers) is essential for privacy and safety; however, existing approaches are unstable and unreliable. A widely used strategy, the gradient difference method, applies gradient descent to retained data while performing gradient ascent on forgotten data. When combined with cross-entropy, this procedure can trigger the unbounded growth of weights and gradients, degrading both forgetting and retention. We provide a theoretical framework that explains this failure by showing how ascent destabilizes optimization in transformer feedforward MLP layers. Guided by this insight, we propose *Bounded Parameter-Efficient Unlearning*, which stabilizes LoRA-based fine-tuning by applying bounded functions to MLP adapters. This controls the weight dynamics during ascent and enables reliable convergence. We validate the approach on Vision Transformer class deletion on CIFAR-100, where GD+Sine is the only evaluated method to achieve both high forget quality and model utility across ViT-B/16, ViT-L/14, and DeiT-S architectures, and demonstrate generality on language-model benchmarks (TOFU, TDEC, MUSE) across architectures from 22M to 8B parameters, achieving improved forgetting while preserving utility.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。