用梯度上升实现可证明的模型删数，不需重训练
Provable Unlearning with Gradient Ascent on Two-Layer ReLU Neural Networks
- 通过梯度上升反向影响特定数据点
- 高维下两层ReLU网络可逼近重训练效果
- 在合成数据上既删数又保持泛化能力
机器删数旨在移除已训练模型中的特定数据，以应对日益增长的隐私与伦理问题。本文对一种简单且广泛使用的方法——梯度上升——进行了理论分析，该方法可在不重新训练的前提下逆转特定数据点的影响。利用梯度下降的隐式偏差，使其趋向于满足边际最大化问题的Karush-Kuhn-Tucker（KKT）条件，我们通过评估模型在保留数据上满足这些条件的程度，量化了删数后模型的质量。为此，我们提出一个新的成功标准，称为$(ε, δ, τ)$-成功删数，并证明：在高维数据下的线性模型和两层神经网络中，适当缩放的梯度上升步长可满足该标准，且其结果模型在保留数据上的表现接近于从头重训练的解。此外，在合成高斯混合数据设置中，梯度上升在成功删数的同时仍能保持模型的泛化性能。
原文摘要 · Abstract (English)
Machine Unlearning aims to remove specific data from trained models, addressing growing privacy and ethical concerns. We provide a theoretical analysis of a simple and widely used method - gradient ascent - used to reverse the influence of a specific data point without retraining from scratch. Leveraging the implicit bias of gradient descent towards solutions that satisfy the Karush-Kuhn-Tucker (KKT) conditions of a margin maximization problem, we quantify the quality of the unlearned model by evaluating how well it satisfies these conditions w.r.t. the retained data. To formalize this idea, we propose a new success criterion, termed \textbf{$(ε, δ, τ)$-successful} unlearning, and show that, for both linear models and two-layer neural networks with high dimensional data, a properly scaled gradient-ascent step satisfies this criterion and yields a model that closely approximates the retrained solution on the retained data. We also show that gradient ascent performs successful unlearning while still preserving generalization in a synthetic Gaussian-mixture setting.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。