通过精选数据删除,高效消除模型中的特定信息分布。
Distributional Machine Unlearning via Selective Data Removal
- 基于统计影响集中原理,挑选少量关键样本实现精准删除。
- 在低差异场景下,采样效率比随机删除高两倍以上。
- 适合需要清除偏见或有害内容的模型可解释性与合规应用。
机器学习系统日益面临移除整个信息域(如有害语言或偏见)的需求,而非单个用户数据。完全删除成本高昂,而随机部分删除又效率低下。我们发现,一个数据域的统计影响往往集中在少数样本中,这为避免无效的部分删除和不必要的全量删除提供了路径。本文提出分布式遗忘(distributional unlearning)框架,旨在选择一个小子集,在遗忘特定分布的同时保留期望分布。利用相对熵约束,我们推导出高斯分布下的精确删改-保留帕累托前沿,并证明在编辑数据上训练的模型能达到相应的对数损失界。我们提出一种基于距离的筛选算法,在挑战性的低差异情形下,采样效率比随机删除高出约两倍。在合成数据、文本(Jigsaw)、图像(CIFAR-10)、短信垃圾邮件(SMS spam)等多类数据集上的实验表明,该方法仅需15%-82%的删除量即可实现强遗忘效果,例如使初始遗忘集准确率减半。结果表明,小规模遗忘集通常已足够,为更高效、严谨的子群体遗忘奠定了基础。
原文摘要 · Abstract (English)
Machine learning systems increasingly face requirements to remove entire domains of information--such as toxic language or biases--rather than individual user data. This task presents a dilemma: full removal of the unwanted domain data is computationally expensive, while random partial removal is statistically inefficient. We find that a domain's statistical influence is often concentrated in a small subset of its data samples, suggesting a path between ineffective partial removal and unnecessary complete removal. We formalize this as distributional unlearning: a framework to select a small subset that balances forgetting an unwanted distribution while preserving a desired one. Using Kullback-Leibler divergence constraints, we derive the exact removal-preservation Pareto frontier for Gaussian distributions and prove that models trained on the edited data achieve corresponding log-loss bounds. We propose a distance-based selection algorithm and show it is quadratically more sample-efficient than random removal in the challenging low-divergence regime. Experiments across synthetic, text, and image datasets (Jigsaw, CIFAR-10, SMS spam) show our method requires 15-82% less deletion than full removal for strong unlearning effects, e.g., halving initial forget set accuracy. Ultimately, by showing a small forget set often suffices, our framework lays the foundations for more scalable and rigorous subpopulation unlearning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。