用动态权重神经元实现大模型知识高效批量修改
Massive Editing for Large Language Models Based on Dynamic Weight Generation
- 在模型层添加动态权重神经元，通过扩散模型按查询生成权重
- 大幅提升知识编辑的可靠性、通用性与局部性，局部性指标提升显著
- 适合需要低成本大规模更新知识的大模型应用者
知识编辑（KE）旨在以较低成本修改大语言模型（LLMs）中的部分知识。当前，在保证编辑的可靠性、通用性和局部性指标的前提下，对大模型进行大规模知识编辑仍具挑战。本文提出基于动态权重生成的大规模知识编辑方法（MeG）。MeG 在 LLM 的特定层附加一个动态权重神经元，并利用扩散模型根据输入查询条件生成该神经元的权重，仅通过增加一个动态权重神经元即可实现大规模知识编辑。实验表明，相较于现有方法，MeG 在可靠性、通用性和局部性指标上均有显著提升，尤其在局部性指标的绝对值上提升明显。代码已开源：https://github.com/RodeWayne/MeG-for-Knowledge-Editing。
原文摘要 · Abstract (English)
Knowledge Editing (KE) is a field that studies how to modify some knowledge in Large Language Models (LLMs) at a low cost (compared to pre-training). Currently, performing large-scale edits on LLMs while ensuring the Reliability, Generality, and Locality metrics of the edits remain a challenge. This paper proposes a Massive editing approach for LLMs based on dynamic weight Generation (MeG). Our MeG involves attaching a dynamic weight neuron to specific layers of the LLMs and using a diffusion model to conditionally generate the weights of this neuron based on the input query required for the knowledge. This allows the use of adding a single dynamic weight neuron to achieve the goal of large-scale knowledge editing. Experiments show that our MeG can significantly improve the performance of large-scale KE in terms of Reliability, Generality, and Locality metrics compared to existing knowledge editing methods, particularly with a high percentage point increase in the absolute value index for the Locality metric, demonstrating the advantages of our proposed method. Code is available at https://github.com/RodeWayne/MeG-for-Knowledge-Editing.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。