通过分步删改机制，让小模型精准更新知识且不遗忘旧信息。
Surgical Knowledge Rewrite in Compact LLMs: An 'Unlearn-then-Learn' Strategy with ($IA^3$) for Localized Factual Modulation and Catastrophic Forgetting Mitigation
- 先定位错误知识所在模块，再针对性重写，避免全模型调整。
- 新知识准确率达98.5%，旧知识遗忘率高达96%，且关键知识几乎不丢失。
- 适合需要安全可控知识更新的医疗、金融等场景使用。
大语言模型在动态知识更新中面临挑战，尤其当新信息与深层嵌入的事实冲突时，易产生难以采纳新事实和严重灾难性遗忘的问题。本文提出一种基于参数高效微调（PEFT）技术的‘先删除后学习’策略，利用$IA^3$方法实现精确知识编辑。该策略首先通过电路定位阶段识别并靶向负责冲突事实的内部组件。在microsoft/Phi-3-mini-4k-instruct上实验表明，该机制驱动的两阶段方法使新事实修改准确率达98.50%，同时有效抑制原冲突事实（遗忘率96.00%）。关键的是，其定位精度达72.00% F_control，显著优于直接微调（约20% F_control），得益于可解释性引导的精准干预。定性分析揭示‘软遗忘’机制：原知识未被彻底删除，仅抑制默认检索，仍可条件性恢复，提升模型安全性与可控性。该研究为紧凑型LLM中的精准、局部化与安全知识管理带来重要进展。
原文摘要 · Abstract (English)
Large Language Models (LLMs) struggle with dynamic knowledge updates, especially when new information conflicts with deeply embedded facts. Such conflicting factual edits often lead to two critical issues: resistance to adopting the new fact and severe catastrophic forgetting of unrelated knowledge. This paper introduces and evaluates a novel "unlearn-then-learn" strategy for precise knowledge editing in LLMs, leveraging the parameter-efficient fine-tuning (PEFT) technique, Infused Adapter by Inhibiting and Amplifying Inner Activations ($IA^3$). Crucially, this two-stage approach is powered by an initial circuit localization phase that identifies and targets the specific internal components responsible for encoding the conflicting fact. Through a rigorous experimental methodology on microsoft/Phi-3-mini-4k-instruct, we demonstrate that this mechanistically informed two-stage approach achieves near-perfect accuracy (98.50%) for the new, modulated fact while simultaneously effectively suppressing the original conflicting fact (96.00% forget rate). Critically, our strategy exhibits unprecedented localization (72.00% F_control accuracy), dramatically mitigating catastrophic forgetting observed in direct fine-tuning approaches (which showed as low as ~20% F_control accuracy), a direct benefit of our targeted interpretability-guided intervention. Furthermore, qualitative analysis reveals a nuanced mechanism of "soft forgetting," where original knowledge is suppressed from default retrieval but remains latent and conditionally accessible, enhancing model safety and control. These findings represent a significant advancement towards precise, localized, and safe knowledge management in compact LLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。