精准删除语言模型中无关信息，提升隐私保护效果。
Unlearning What Matters: Token-Level Attribution for Precise Language Model Unlearning
- 按词元重要性识别关键信息，只对相关部分更新。
- 在三个模型上验证，遗忘效果更好且模型性能更稳。
- 适合需要精准删除数据的AI安全与合规场景。
机器遗忘已成为应对大语言模型（LLMs）隐私、安全与监管问题的关键能力。现有方法在序列层面统一更新所有词元，尽管仅部分词元编码了需删除的知识，导致梯度噪声增加、性能下降，遗忘效果不佳。我们提出TokenUnlearn，一种基于词元级别的归因框架，通过掩码和熵信号结合，生成词元重要性评分，实现精准词元选择。开发两种互补策略：硬选择（仅对高重要性词元进行遗忘）和软加权（根据重要性调节梯度贡献），均能扩展现有方法至词元级别。理论分析表明，词元级选择可提升梯度信噪比。在TOFU和WMDP基准上，跨三种模型架构的实验表明，相比序列级基线，该方法在遗忘有效性和性能保留方面均有持续提升。
原文摘要 · Abstract (English)
Machine unlearning has emerged as a critical capability for addressing privacy, safety, and regulatory concerns in large language models (LLMs). Existing methods operate at the sequence level, applying uniform updates across all tokens despite only a subset encoding the knowledge targeted for removal. This introduces gradient noise, degrades utility, and leads to suboptimal forgetting. We propose TokenUnlearn, a token-level attribution framework that identifies and selectively targets critical tokens. Our approach combines knowledge-aware signals via masking, and entropy-aware signals to yield importance scores for precise token selection. We develop two complementary strategies: hard selection, applying unlearning only to high-importance tokens, and soft weighting, modulating gradient contributions based on importance scores. Both extend existing methods to token-level variants. Theoretical analysis shows token-level selection improves gradient signal-to-noise ratio. Experiments on TOFU and WMDP benchmarks across three model architectures demonstrate consistent improvements over sequence-level baselines in both forgetting effectiveness and utility preservation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。