让大模型遗忘特定知识，同时保持回答能力并可控回退。
LLM Unlearning via Neural Activation Redirection
- 通过激活重定向技术，将需遗忘数据的表示引导至无法作答的区域。
- 在多个模型上实现2.9到11.7倍的综合性能提升，且响应依然连贯恰当。
- 仅需单个降维矩阵，效率提升20倍，支持持续遗忘与抗攻击场景。
从大语言模型中选择性移除知识极具价值。现有方法常难以平衡遗忘效果与模型可用性，且缺乏推理时控制以模拟从未见过该数据的状态。本文提出LUNAR，基于线性表征假设，通过将需遗忘数据的表示重定向至表达“无法作答”的激活区域来实现遗忘。我们证明对比特征并非激活重定向的有效前提，而LUNAR在多种基模型上达到当前最优遗忘性能，并具备更优可控性。具体而言，其在组合遗忘效能与模型效用得分（偏差分）上实现2.9至11.7倍提升；遗忘后仍能生成连贯、符合语境的回答。此外，LUNAR将参数更新压缩为单一降维矩阵，效率提升20倍，鲁棒性显著增强。最后，实验表明其对白盒对抗攻击具有鲁棒性，且适用于实际场景中的序列化遗忘请求。
原文摘要 · Abstract (English)
The ability to selectively remove knowledge from LLMs is highly desirable. However, existing methods often struggle with balancing unlearning efficacy and retain model utility, and lack controllability at inference time to emulate base model behavior as if it had never seen the unlearned data. In this paper, we propose LUNAR, a novel unlearning method grounded in the Linear Representation Hypothesis and operates by redirecting the representations of unlearned data to activation regions that expresses its inability to answer. We show that contrastive features are not a prerequisite for effective activation redirection, and LUNAR achieves state-of-the-art unlearning performance and superior controllability. Specifically, LUNAR achieves between 2.9x and 11.7x improvement in the combined unlearning efficacy and model utility score (Deviation Score) across various base models and generates coherent, contextually appropriate responses post-unlearning. Moreover, LUNAR effectively reduces parameter updates to a single down-projection matrix, a novel design that significantly enhances efficiency by 20x and robustness. Finally, we demonstrate that LUNAR is robust to white-box adversarial attacks and versatile in real-world scenarios, including handling sequential unlearning requests.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。