arXiv:2506.10946cs.LGcs.AI2025-06被引 1

让大模型删数据时不伤及有用知识,还能保护隐私。

GUARD: Guided Unlearning and Retention via Data Attribution for Large Language Models

  • 用数据归属度评分指导删数据,精准控制遗忘范围。
  • 删10%数据时,保留集准确率提升194.92%,知识保留增16.20%。
  • 适合需要合规删数据又不想损失性能的模型应用者。

大型语言模型中的去学习(unlearning)因法规合规、版权保护和隐私问题日益重要。但现有方法常因删除特定数据而意外损害模型性能,即产生非预期遗忘。以往研究多关注架构改进,却忽视了数据层面因素的影响。为此,我们提出GUARD框架,通过轻量级代理数据归属度指标衡量需删与需留数据的对齐程度,设计自适应非均匀遗忘权重——权重与归属度成反比,实现遗忘力量的智能分配,有效缓解无意识的知识丢失。理论分析表明,该方法显著提升保留性能,同时保持与前沿方法相当的遗忘效果。在TOFU与MUSE基准测试中,针对多种大模型架构的实验显示:当删除10%训练数据时,TOFU保留集真值率提升最高达194.92%,MUSE NEWS保留集知识保留率提高16.20%,且隐私泄露增长可控,优于当前最优方法。

原文摘要 · Abstract (English)

Unlearning in large language models is becoming increasingly important due to regulatory compliance, copyright protection, and privacy concerns. However, a key challenge in LLM unlearning is unintended forgetting, where the removal of specific data inadvertently impairs the utility of the model and its retention of valuable, desired information. While prior work has primarily focused on architectural innovations, the influence of data-level factors on unlearning performance remains underexplored. As a result, existing methods often suffer from degraded retention when forgetting high-impact data. To address this problem, we propose GUARD, a novel framework for Guided Unlearning And Retention via Data attribution. At its core, GUARD introduces a lightweight proxy data attribution metric tailored for LLM unlearning, which quantifies the alignment between the Forget and Retain sets while remaining computationally efficient. Building on this, we design a novel unlearning objective that assigns adaptive, nonuniform unlearning weights to samples, inversely proportional to their proxy attribution scores. Through such a reallocation of unlearning power, GUARD mitigates unintended retention loss. We also provide rigorous theoretical guarantees that GUARD significantly improves retention while maintaining forgetting metrics comparable to prior methods. Extensive experiments on the TOFU and MUSE benchmarks across multiple LLM architectures demonstrate that GUARD reduces utility sacrifice on the TOFU Retain Set by up to 194.92 percent in terms of Truth Ratio when forgetting 10 percent of the training data, and improves knowledge retention on the MUSE NEWS Retain Set by 16.20 percent, with comparable or very moderate increases in privacy loss compared to state-of-the-art methods.

大模型去学习知识保留数据隐私

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。