让语言模型主动遗忘特定概念，保持其他能力不变。
Erasing Conceptual Knowledge from Language Models
- 用模型自身判断能力识别并降低特定概念的生成概率。
- 在生物安全、网络安全等任务上接近随机表现，仍保持生成连贯性。
- 适合需要可控知识删除的应用，如隐私保护与安全合规。
本文提出一种名为语言记忆擦除（ELM）的概念级去学习方法，其核心思想是利用模型自身的内省分类能力，通过匹配分布来实现针对性的知识删除。我们发现，有效的去学习应依赖模型对自身知识的评估能力，即使用语言模型本身作为分类器，识别并降低与不希望保留概念相关的生成概率。ELM在此框架下实施低秩更新，精准削弱特定概念相关内容的生成概率，同时保持模型整体能力不受影响。实验表明，在生物安全、网络安全及文学领域等擦除任务中，经ELM处理的模型在目标概念测试上表现接近随机，但生成内容依然连贯，且在无关任务上的基准性能未下降，同时具备较强的抗对抗攻击鲁棒性。代码、数据及训练好的模型已公开于https://elm.baulab.info。
原文摘要 · Abstract (English)
In this work, we introduce Erasure of Language Memory (ELM), a principled approach to concept-level unlearning that operates by matching distributions defined by the model's own introspective classification capabilities. Our key insight is that effective unlearning should leverage the model's ability to evaluate its own knowledge, using the language model itself as a classifier to identify and reduce the likelihood of generating content related to undesired concepts. ELM applies this framework to create targeted low-rank updates that reduce generation probabilities for concept-specific content while preserving the model's broader capabilities. We demonstrate ELM's efficacy on biosecurity, cybersecurity, and literary domain erasure tasks. Comparative evaluation reveals that ELM-modified models achieve near-random performance on assessments targeting erased concepts, while simultaneously preserving generation coherence, maintaining benchmark performance on unrelated tasks, and exhibiting strong robustness to adversarial attacks. Our code, data, and trained models are available at https://elm.baulab.info
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。