通过压缩表示空间实现可靠遗忘,提升隐私保护效率
Representation Unlearning: Forgetting through Information Compression
- 在模型表示层构建信息瓶颈,直接压缩需遗忘数据的特征
- 相比参数修改方法,遗忘更彻底且保留有用性能更好
- 适用于有无保留数据的两种场景,计算更高效
机器遗忘旨在消除特定训练数据对模型的影响,这一需求源于隐私法规与模型鲁棒性要求。现有方法通常修改模型参数,但此类更新易不稳定、计算成本高,且受限于局部近似。我们提出表示遗忘(Representation Unlearning)框架,直接在模型的表示空间中执行遗忘操作。不修改参数,而是学习一种对表示的变换,建立信息瓶颈:最大化与保留数据的互信息,同时抑制与待遗忘数据的信息。我们推导出可计算的变分近似,并展示了其在两种实用场景中的实现:当保留与遗忘数据均可用时,以及仅能访问遗忘数据的零样本场景。在多个基准上的实验表明,该方法在遗忘可靠性、性能保留和计算效率方面均优于以参数为中心的基线方法。
原文摘要 · Abstract (English)
Machine unlearning seeks to remove the influence of specific training data from a model, a need driven by privacy regulations and robustness concerns. Existing approaches typically modify model parameters, but such updates can be unstable, computationally costly, and limited by local approximations. We introduce Representation Unlearning, a framework that performs unlearning directly in the model's representation space. Instead of modifying model parameters, we learn a transformation over representations that imposes an information bottleneck: maximizing mutual information with retained data while suppressing information about data to be forgotten. We derive variational surrogates that make this objective tractable and show how they can be instantiated in two practical regimes: when both retain and forget data are available, and in a zero-shot setting where only forget data can be accessed. Experiments across several benchmarks demonstrate that Representation Unlearning achieves more reliable forgetting, better utility retention, and greater computational efficiency than parameter-centric baselines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。