arXiv:2511.17100cs.LGcs.AI2025-11被引 5

提出几何解耦遗忘方法,减少大模型遗忘时对保留知识的副作用。

Geometric-disentangelment Unlearning

  • 基于参数更新方向与保留梯度正交性,实现无副作用遗忘。
  • 在多个数据集上提升遗忘强度62%,保留知识漂移降低31%。
  • 轻量级插件设计,可无缝接入现有遗忘方法,适合模型安全研究者。

大语言模型可能内化隐私或有害内容,促使遗忘机制移除特定数据集的同时保留原有知识。然而,现有遗忘更新常导致保留知识的附带退化,形成持久权衡。现有方法多为启发式,理论方法依赖离线特征构建,无法捕捉模型更新时的遗忘-保留交互。本文从第一性原理出发,将“无副作用”形式化为小参数更新下的保留不变性,并证明在优化器诱导几何下,保留损失局部不变当且仅当更新方向与保留梯度张成子空间正交。基于此,提出几何解耦遗忘(GU),一种轻量级、理论严谨的投影机制,可即插即用至现有基于梯度的遗忘方法中,缓解遗忘-保留副作用。在TOFU、MUSE和WMDP-cyber数据集上的实验表明,GU增强了遗忘效果同时降低了保留知识漂移。当集成至SimNPO时,遗忘提取强度(ES)提升最高达62%,保留ES提高31%。代码已开源于https://github.com/Lemutisme/Geometric-Unlearning。

原文摘要 · Abstract (English)

Large language models (LLMs) can internalize private or harmful content, motivating unlearning that removes a forget set while preserving retaining knowledge. However, forgetting updates often cause collateral degradation on retaining knowledge, creating a persistent trade-off. Existing LLM unlearning methods are often heuristic, and other theoretical approaches rely on offline feature constructions that do not capture update-time forget-retain interaction in LLMs. To address this limitation, we aim to develop an LLM unlearning method that reduces the forget-retain trade-off with theoretical guarantees. We take a first-principles view by formalizing "no side effects" as local retain invariance under small parameter updates, and prove an equivalence under optimizer-induced geometry: the retain loss is locally invariant if and only if the update direction is orthogonal to the subspace spanned by retain gradients. Based on the insight, we propose Geometric-disentanglement Unlearning (GU), a lightweight and theoretically grounded projection that can be plug-and-play to existing gradient-based unlearning methods to mitigate forget-retain side effects. Experiments on TOFU, MUSE, and WMDP-cyber show that GU strengthens forgetting while reducing retain drift. When added to SimNPO, it achieves up to 62\% improved forgetting Extraction Strength (ES) and 31\% higher retain ES. We open-sourced our code in https://github.com/Lemutisme/Geometric-Unlearning.

大模型遗忘机制几何方法

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。