用词熵值筛选要删的词,删得准还保有用。
Forget What Matters, Keep the Rest: Selective Unlearning of Informative Tokens

- 用预测分布熵衡量词的重要程度,高熵词更可能被删。
- 在不降性能前提下,比现有方法多删30%无关内容。
- 适合需要精准擦除数据的模型安全场景。
大语言模型中的去学习技术可防范恶意行为。传统方法对所有词一视同仁地施加遗忘损失,易导致模型能力下降。现有研究虽尝试按词重要性加权,但依赖人工标注或外部语法分析工具,难以捕捉上下文和模型自身状态。本文提出熵引导的词级遗忘正则化(ETW):基于预测分布熵判断词的语义重要性——信息量大的词通常具有更高熵,而功能词如“the”熵值较低且可预测。实验表明,该方法在保留模型性能的同时,能更有效实现去学习,优于现有词级方法。
原文摘要 · Abstract (English)
Unlearning in large language models (LLMs) has emerged as a promising safeguard against adversarial behaviors. When the forgetting loss is applied uniformly without considering token-level semantic importance, model utility can be unnecessarily degraded. Recent studies have explored token-wise loss regularizers that prioritize informative tokens, but largely rely on ground-truth confidence or external linguistic parsers, which limits their ability to capture contextual information or the model's overall predictive state. Intuitively, function words like "the" primarily serve syntactic roles and are highly predictable with little ambiguity, but informative words admit multiple plausible alternatives with greater uncertainty. Based on this intuition, we propose Entropy-guided Token Weighting (ETW), a token-level unlearning regularizer that uses entropy of the predictive distribution as a proxy for token informativeness. We demonstrate that informative tokens tend to have higher entropy, whereas structural tokens tend to have lower entropy. This behavior enables ETW to achieve more effective unlearning while better preserving model utility than existing token-level approaches.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。