arXiv:2509.15631cs.CLcs.LG2025-09被引 3

让大模型真正忘记特定知识,通过改造内部激活实现无损擦除。

Sparse-Autoencoder-Guided Internal Representation Unlearning for Large Language Models

  • 在稀疏自编码器空间中重构目标知识的内部激活,使其与未知实体对齐。
  • 实验表明该方法能有效消除目标知识的激活信号,且不损伤其他知识。
  • 适合需要精准删除隐私或版权内容的场景,避免模型崩溃。

随着大语言模型在各类应用中普及,隐私与版权问题凸显,亟需更有效的模型遗忘技术。现有方法多通过额外训练(如梯度上升)抑制不良输出,但仅压制响应而未清除模型内部存储的知识,且易引发模型崩溃。为此,本文提出一种新遗忘方法,直接干预模型内部激活。将遗忘定义为:目标实体的激活状态与‘未知’实体无法区分。该方法在稀疏自编码器潜空间中,将目标实体的激活从已知实体拉远、向未知实体靠拢。通过使模型将目标识别为‘未知’,实现真正遗忘,避免过度抑制和模型崩溃。实验证明,该方法能有效对齐被遗忘目标的内部激活,而传统抑制方法难以达成此效果;同时在问答任务中显著降低目标知识召回率,且对非目标知识影响极小。

原文摘要 · Abstract (English)

As large language models (LLMs) are increasingly deployed across various applications, privacy and copyright concerns have heightened the need for more effective LLM unlearning techniques. Many existing unlearning methods aim to suppress undesirable outputs through additional training (e.g., gradient ascent), which reduces the probability of generating such outputs. While such suppression-based approaches can control model outputs, they may not eliminate the underlying knowledge embedded in the model's internal activations; muting a response is not the same as forgetting it. Moreover, such suppression-based methods often suffer from model collapse. To address these issues, we propose a novel unlearning method that directly intervenes in the model's internal activations. In our formulation, forgetting is defined as a state in which the activation of a forgotten target is indistinguishable from that of ``unknown'' entities. Our method introduces an unlearning objective that modifies the activation of the target entity away from those of known entities and toward those of unknown entities in a sparse autoencoder latent space. By aligning the target's internal activation with those of unknown entities, we shift the model's recognition of the target entity from ``known'' to ``unknown'', achieving genuine forgetting while avoiding over-suppression and model collapse. Empirically, we show that our method effectively aligns the internal activations of the forgotten target, a result that the suppression-based approaches do not reliably achieve. Additionally, our method effectively reduces the model's recall of target knowledge in question-answering tasks without significant damage to the non-target knowledge.

大模型遗忘学习内部表示隐私保护

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。