在过参数化场景下,提出新方法实现高效模型删样。
Machine Unlearning under Overparameterization
- 定义最小复杂度插值解作为删样目标,突破传统梯度扰动局限。
- 仅需原模型在保留数据上的梯度,通过正交约束优化实现删样。
- 在多种模型上证明理论有效性,实验性能超越现有基线。
机器删样算法旨在消除特定训练样本的影响,理想情况下恢复仅基于剩余数据训练的模型。本文研究过参数化设置下的删样问题,此时存在多个能插值数据的模型,若仍沿用欠参数化情形中以损失最小化为目标的定义,则不再适用,因为原始模型可能已插值于保留数据集且满足该条件。在此情形下,损失梯度消失,导致依赖梯度扰动的已有方法失效,因而需要新的删样定义与算法。本文将删样解定义为保留数据上的最小复杂度插值器,并提出一种仅需访问原始解在保留数据集上的模型梯度的新算法框架。通过在与这些梯度正交的扰动空间中最小化带正则化的目标函数,实现对插值条件的一阶松弛。针对不同模型类别,提供精确与近似删样保证,并实证表明该框架在各类删样实验中均优于现有基线方法。
原文摘要 · Abstract (English)
Machine unlearning algorithms aim to remove the influence of specific training samples, ideally recovering the model that would have resulted from training on the remaining data alone. We study unlearning in the overparameterized setting, where many models interpolate the data, and defining the solution as any loss minimizer over the retained set$\unicode{x2013}$as in prior work in the underparameterized setting$\unicode{x2013}$is inadequate, since the original model may already interpolate the retained data and satisfy this condition. In this regime, loss gradients vanish, rendering prior methods based on gradient perturbations ineffective, motivating both new unlearning definitions and algorithms. For this setting, we define the unlearning solution as the minimum-complexity interpolator over the retained data and propose a new algorithmic framework that only requires access to model gradients on the retained set at the original solution. We minimize a regularized objective over perturbations constrained to be orthogonal to these model gradients, a first-order relaxation of the interpolation condition. For different model classes, we provide exact and approximate unlearning guarantees and demonstrate that an implementation of our framework outperforms existing baselines across various unlearning experiments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。