用轻量级网络精准编辑大模型1万条知识，还能保持模型整体能力。
Latent Knowledge Scalpel: Precise and Massive Knowledge Editing for Large Language Models
- 通过轻量超网络操控隐藏表示，实现对特定实体知识的精准替换。
- 在Llama-2和Mistral上成功完成1万条知识同时编辑，效果显著。
- 适合需要大规模知识更新但又不想损伤模型通用能力的研究者。
大语言模型（LLMs）常因预训练数据包含错误或过时信息，在推理时产生不准确或有偏见的输出。现有模型编辑方法难以同时处理大量事实信息的修改，且可能损害模型的通用能力。本文实证研究表明，可通过类似自然语言输入编辑的方式，修改大模型内部表示中的实体信息。基于此，我们提出潜知识手术刀（Latent Knowledge Scalpel, LKS），一种利用轻量级超网络操纵特定实体隐含知识的编辑方法，实现精确且大规模的知识更新。在Llama-2和Mistral上的实验表明，即使同时编辑多达10,000条知识，LKS仍能有效完成知识修正，同时保持模型的通用能力。代码已公开：https://github.com/Linuxin-xxx/LKS。
原文摘要 · Abstract (English)
Large Language Models (LLMs) often retain inaccurate or outdated information from pre-training, leading to incorrect predictions or biased outputs during inference. While existing model editing methods can address this challenge, they struggle with editing large amounts of factual information simultaneously and may compromise the general capabilities of the models. In this paper, our empirical study demonstrates that it is feasible to edit the internal representations of LLMs and replace the entities in a manner similar to editing natural language inputs. Based on this insight, we introduce the Latent Knowledge Scalpel (LKS), an LLM editor that manipulates the latent knowledge of specific entities via a lightweight hypernetwork to enable precise and large-scale editing. Experiments conducted on Llama-2 and Mistral show even with the number of simultaneous edits reaching 10,000, LKS effectively performs knowledge editing while preserving the general abilities of the edited LLMs. Code is available at: https://github.com/Linuxin-xxx/LKS.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。