精准删除大模型中的特定概念知识，效果优于现有方法。
Precise In-Parameter Concept Erasure in Large Language Models
- 通过分解MLP特征并定位目标概念方向，直接修改参数实现精准擦除。
- 在多个概念上将目标准确率降至7.7%，擦除特异性提升31%。
- 适合需要安全可控知识删除的AI部署场景，如隐私保护与合规应用。
大型语言模型在预训练中常习得敏感信息或受版权保护的内容，这些内容在下游应用中可能带来风险。现有方法依赖微调、低秩适配器或事实级编辑，但存在粒度粗、深度浅或无效等问题。本文提出PISCES（Precise In-parameter Suppression for Concept EraSure），一种通过直接编辑参数空间中编码概念的方向来精确擦除整个概念的新框架。PISCES利用解耦模型将MLP向量分解为可解释特征，结合自动化可解释性技术识别与目标概念相关的特征，并将其从模型参数中移除。在Gemma 2和Llama 3.1上对多种概念进行实验表明，PISCES在有效性上略优于领先方法，将目标概念准确率降低至7.7%，同时擦除特异性提升最高达31%，鲁棒性提升最高达38%。结果证明，基于特征的参数内编辑能更精确、可靠地清除语言模型中的概念知识。
原文摘要 · Abstract (English)
Large language models (LLMs) often acquire knowledge during pretraining that is undesirable in downstream deployments, e.g., sensitive information or copyrighted content. Existing approaches for removing such knowledge rely on fine-tuning, training low-rank adapters or fact-level editing, but these are either too coarse, too shallow, or ineffective. In this work, we propose PISCES (Precise In-parameter Suppression for Concept EraSure), a novel framework for precisely erasing entire concepts from model parameters by directly editing directions that encode them in parameter space. PISCES uses a disentangler model to decompose MLP vectors into interpretable features, identifies those associated with a target concept using automated interpretability techniques, and removes them from model parameters. Experiments on Gemma 2 and Llama 3.1 over various concepts show that PISCES achieves modest gains in efficacy over leading erasure methods, reducing accuracy on the target concept to as low as 7.7%, while dramatically improving erasure specificity (by up to 31%) and robustness (by up to 38%). Overall, these results demonstrate that feature-based in-parameter editing enables a more precise and reliable approach for removing conceptual knowledge in language models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。