不改模型权重,用输入自适应的激活重定向实现推理时遗忘。
Inference-Time Machine Unlearning via Gated Activation Redirection
- 基于输入动态调整激活,通过保留范数的旋转实现无损干预。
- 在三个模型规模上超越12种基线方法,且在量化后仍有效。
- 适合需要持续遗忘、部署时低开销的场景,如隐私保护应用。
大型语言模型会记忆海量训练数据,引发隐私、版权和安全问题。机器遗忘旨在移除特定遗忘集的影响,同时保持模型性能,理想情况下等价于从头训练不含遗忘集的模型。现有方法通过梯度更新参数实现,但计算成本高、权重修改不可逆,且在量化后性能下降。近期替代方案是激活工程:在推理时改变激活以引导模型行为。然而,简单激活引导对所有输入使用全局干预向量,导致行为意外改变。本文提出推理时遗忘方法 GUARD-IT,无需训练和梯度,通过输入依赖的激活重定向实现遗忘。干预以残差流中的范数保持旋转形式实施,不修改模型权重。在 TOFU 与 MUSE 数据集上的实验表明,GUARD-IT 在三种模型规模下均匹配或超过12种基于梯度的基线方法,且是唯一在所有设置中同时保持性能、抑制记忆并避免灾难性崩溃的方法。此外,该方法支持无需重训的持续遗忘,并在量化场景中依然有效,而参数编辑方法在此类场景下性能退化。
原文摘要 · Abstract (English)
Large Language Models memorize vast amounts of training data, raising concerns regarding privacy, copyright infringement, and safety. Machine unlearning seeks to remove the influence of a targeted forget set while preserving model performance, ideally approximating a model retrained from scratch without the forget set. Existing approaches aim to achieve this by updating model parameters via gradient-based methods. However, these updates are computationally expensive, lead to irreversible weight changes, and degrade when the model is quantized for deployment. A recent alternative to changing model weights is activation engineering, where activations are changed during inference to steer model behavior. Despite circumventing weight editing, naive activation steering introduces its own failure modes, as a single global steering vector applies the same intervention to every input, leading to unintended changes in model behavior. We introduce Inference-Time Unlearning via Gated Activation Redirection (GUARD-IT), a training- and gradient-free method that unlearns via input-dependent activation steering at inference time. The resulting intervention is applied as a norm-preserving rotation in the residual stream, leaving model weights untouched. Experiments on TOFU and MUSE show that GUARD-IT matches or exceeds 12 gradient-based baselines across three model scales, while being the only method to simultaneously preserve utility, suppress memorization, and avoid catastrophic collapse across all settings. GUARD-IT further supports continual unlearning without retraining, and remains effective under quantization, a scenario in which parameter-editing methods degrade.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。