提出正交梯度方法，让模型在无完整训练集时仍能高效清除特定数据影响。
Go Beyond Your Means: Unlearning with Per-Sample Gradient Orthogonalization
- 将删除数据的梯度投影到保留数据梯度的正交空间，避免冲突。
- 在语音识别等任务上优于现有方法，保持模型性能不下降。
- 适用于训练集不可用或仅可访问小部分数据的场景。
机器遗忘旨在模型训练后消除某些问题数据的影响。主要挑战在于有效移除指定数据的同时，不损害模型在剩余数据上的整体性能。现有方法通常通过平衡‘删除’数据的梯度上升与‘保留’数据集的梯度下降来应对，但许多情况下训练数据无法获取——模型发布时通常不附带训练集，且用户可能仅能访问一小部分训练数据。为此，我们提出OrthoGrad，一种新方法：不采用竞争性上升与下降过程，而是通过将删除数据的梯度投影至保留批次所有梯度所张成子空间的正交方向，有效规避梯度干扰。我们在多个机器遗忘基准测试中验证了OrthoGrad的有效性，包括自动语音识别任务，结果优于现有方法。
原文摘要 · Abstract (English)
Machine unlearning aims to remove the influence of problematic training data after a model has been trained. The primary challenge in machine unlearning is ensuring that the process effectively removes specified data without compromising the model's overall performance on the remaining dataset. Many existing machine unlearning methods address this challenge by carefully balancing gradient ascent on the `unlearn' data with the gradient descent on a `retain' set that represents the training data. However, in many cases the training dataset is not fully available when we wish to unlearn some concepts, because models are released without their training datasets, and one may only have access to a $\textit{small part of a training set}$. Here, we propose OrthoGrad, a novel approach that mitigates interference between the unlearn set and a small retain set rather than competing ascent and descent processes. Our method projects the gradient of the unlearn set onto the subspace orthogonal to all gradients in the retain batch, effectively avoiding any gradient interference. We demonstrate the effectiveness of OrthoGrad on multiple machine unlearning benchmarks, including automatic speech recognition, outperforming competing methods.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。