提出可证明的机器遗忘方法,区分分布内/外数据的遗忘效率
The Utility and Complexity of in- and out-of-Distribution Machine Unlearning
- 用输出扰动的优化方法实现分布内数据的高效遗忘
- 分布外数据遗忘耗时可能超过重新训练,单样本也如此
- 新梯度下降法保证遗忘速度不降且不损失模型性能
机器遗忘——从已训练模型中选择性移除数据——对解决部署后的隐私问题和知识缺口日益重要。然而现有方法多为启发式,缺乏形式化保障。本文分析了近似遗忘在效用、时间和空间复杂度之间的基本权衡,提供类似差分隐私的严格认证。对于与保留数据分布相似的遗忘数据(分布内),我们证明一种简单通用的方法——带输出扰动的经验风险最小化——能实现紧致的遗忘-效用-复杂度权衡,填补了此前关于差分隐私无法实现‘免费遗忘’的理论空白。但该方法在与保留数据差异显著的遗忘数据(分布外)上失效,此时遗忘时间复杂度可能超过重新训练,即使仅删除一个样本。为此,我们提出一种新的鲁棒且带噪声的梯度下降变体,可证明地摊销遗忘时间复杂度,同时不损害模型效用。
原文摘要 · Abstract (English)
Machine unlearning, the process of selectively removing data from trained models, is increasingly crucial for addressing privacy concerns and knowledge gaps post-deployment. Despite this importance, existing approaches are often heuristic and lack formal guarantees. In this paper, we analyze the fundamental utility, time, and space complexity trade-offs of approximate unlearning, providing rigorous certification analogous to differential privacy. For in-distribution forget data -- data similar to the retain set -- we show that a surprisingly simple and general procedure, empirical risk minimization with output perturbation, achieves tight unlearning-utility-complexity trade-offs, addressing a previous theoretical gap on the separation from unlearning "for free" via differential privacy, which inherently facilitates the removal of such data. However, such techniques fail with out-of-distribution forget data -- data significantly different from the retain set -- where unlearning time complexity can exceed that of retraining, even for a single sample. To address this, we propose a new robust and noisy gradient descent variant that provably amortizes unlearning time complexity without compromising utility.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。