arXiv:2502.07587cs.LG2025-02ICML被引 14

用奇异值分解实现高效删数据,不依赖原始训练集。

SEMU: Singular Value Decomposition for Efficient Machine Unlearning

  • 通过奇异值分解构建低维投影,只改少量参数即可删去特定数据。
  • 实验表明只需修改1%参数就能有效遗忘,且无需原始训练数据。
  • 适合需要合规删数据的AI系统,尤其在隐私保护场景下优势明显。

近年来,生成式基础模型能力飞速发展,但防止有害和不安全行为的方法仍不成熟。在人工智能安全领域,机器遗忘(MU)已成为应对未来安全法规的关键挑战。现有大多数方法聚焦于修改模型中最重要的参数,但通常需微调大量模型参数,导致计算成本高且训练不稳定,通常依赖原始训练数据来缓解。本文提出一种新方法:基于奇异值分解(SVD)创建紧凑的低维投影,实现选择性遗忘特定数据点。我们设计了名为SEMU(Singular Value Decomposition for Efficient Machine Unlearning)的新框架,从两个关键方面优化机器遗忘:第一,大幅减少需修改的模型参数数量,仅做最小权重调整即可移除有害知识;第二,消除对原始训练数据的依赖,无需额外数据即可保留原有知识。大量实验证明,SEMU在性能上具有竞争力,同时在数据使用和修改参数数量上显著提升效率。

原文摘要 · Abstract (English)

While the capabilities of generative foundational models have advanced rapidly in recent years, methods to prevent harmful and unsafe behaviors remain underdeveloped. Among the pressing challenges in AI safety, machine unlearning (MU) has become increasingly critical to meet upcoming safety regulations. Most existing MU approaches focus on altering the most significant parameters of the model. However, these methods often require fine-tuning substantial portions of the model, resulting in high computational costs and training instabilities, which are typically mitigated by access to the original training dataset. In this work, we address these limitations by leveraging Singular Value Decomposition (SVD) to create a compact, low-dimensional projection that enables the selective forgetting of specific data points. We propose Singular Value Decomposition for Efficient Machine Unlearning (SEMU), a novel approach designed to optimize MU in two key aspects. First, SEMU minimizes the number of model parameters that need to be modified, effectively removing unwanted knowledge while making only minimal changes to the model's weights. Second, SEMU eliminates the dependency on the original training dataset, preserving the model's previously acquired knowledge without additional data requirements. Extensive experiments demonstrate that SEMU achieves competitive performance while significantly improving efficiency in terms of both data usage and the number of modified parameters.

机器遗忘奇异值分解隐私保护高效训练

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。