arXiv:2410.03833cs.LGstat.ML2024-10被引 4

提出新方法让模型真正忘记数据,同时更好保留有用信息。

Understanding Fine-tuning in Approximate Unlearning: A Theoretical Perspective

  • 基于保留数据构建权重敏感图,反向优化遗忘效果。
  • 理论证明新方法在遗忘准确率和保留准确率上均显著提升。
  • 适合关注隐私保护与模型可解释性的研究人员。

机器遗忘已成为重要研究方向,旨在从训练好的模型中‘移除’特定数据子集。微调(FT)方法是近似遗忘的核心手段,能有效维持模型性能。然而,现有方法普遍存在无法真正遗忘目标数据的问题。本文首次在线性回归框架下对微调方法进行理论分析,揭示尽管微调模型可实现零剩余损失,但因预训练模型的影响未被充分消除,仍无法真正遗忘。为此,我们提出一种基于保留数据的掩码策略(RBM),通过构建基于剩余数据的权重敏感图,克服传统方法聚焦遗忘数据的局限。理论分析表明,RBM不仅显著提升遗忘准确率(UA),还能保证更高的保留准确率(RA),有效保留遗忘与剩余数据间的共享特征。合成与真实数据集上的实验验证了理论结论,RBM在平衡遗忘准确率、保留准确率及差异度量方面优于现有掩码方法。

原文摘要 · Abstract (English)

Machine Unlearning has emerged as a significant area of research, focusing on `removing' specific subsets of data from a trained model. Fine-tuning (FT) methods have become one of the fundamental approaches for approximating unlearning, as they effectively retain model performance. However, it is consistently observed that naive FT methods struggle to forget the targeted data. In this paper, we present the first theoretical analysis of FT methods for machine unlearning within a linear regression framework, providing a deeper exploration of this phenomenon. Our analysis reveals that while FT models can achieve zero remaining loss, they fail to forget the forgetting data, as the pretrained model retains its influence and the fine-tuning process does not adequately mitigate it. To address this, we propose a novel Retention-Based Masking (RBM) strategy that constructs a weight saliency map based on the remaining dataset, unlike existing methods that focus on the forgetting dataset. Our theoretical analysis demonstrates that RBM not only significantly improves unlearning accuracy (UA) but also ensures higher retaining accuracy (RA) by preserving overlapping features shared between the forgetting and remaining datasets. Experiments on synthetic and real-world datasets validate our theoretical insights, showing that RBM outperforms existing masking approaches in balancing UA, RA, and disparity metrics.

机器遗忘微调隐私保护理论分析

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。