只让模型忘记敏感前缀，不伤通用能力。
Maximizing Local Entropy Where It Matters: Prefix-Aware Localized LLM Unlearning
- 按敏感前缀位置局部最大化熵，精准控制遗忘范围。
- 仅对前k个预测结果去相关，保持核心性能不变。
- 适合需要安全遗忘又怕模型变弱的场景。
机器遗忘旨在让大语言模型遗忘敏感知识的同时保持通用能力。现有方法通常对响应中所有词元一视同仁，在整个词汇表上强制不确定性，导致不必要的性能下降，并将优化扩展到无关区域。为此，我们提出PALU（Prefix-Aware Localized Unlearning），一种基于时间与词汇维度局部熵最大化的框架。PALU发现：(i) 仅抑制敏感前缀即可切断因果生成链；(ii) 仅对前k个logits进行平坦化就足以在关键子空间最大化不确定性。该机制避免了在整个词汇表和参数空间上的冗余优化，同时最小化对通用性能的副作用。全面评估表明，相比最先进基线，PALU在遗忘效果和性能保留方面均表现更优。代码已开源：https://github.com/nxZhai/PALU。
原文摘要 · Abstract (English)
Machine unlearning aims to forget sensitive knowledge from Large Language Models (LLMs) while maintaining general utility. However, existing approaches typically treat all tokens in a response indiscriminately and enforce uncertainty over the entire vocabulary. This global treatment results in unnecessary utility degradation and extends optimization to content-agnostic regions. To address these limitations, we propose PALU (Prefix-Aware Localized Unlearning), a framework driven by a local entropy maximization objective across both temporal and vocabulary dimensions. PALU reveals that (i) suppressing the sensitive prefix alone is sufficient to sever the causal generation link, and (ii) flattening only the top-$k$ logits is adequate to maximize uncertainty in the critical subspace. These findings allow PALU to alleviate redundant optimization across the full vocabulary and parameter space while minimizing collateral damage to general model performance. Comprehensive evaluations validate that PALU achieves superior forgetting efficacy and utility preservation compared to state-of-the-art baselines. Our code is available at https://github.com/nxZhai/PALU.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。