解决大模型批量编辑时同主题冲突问题，让知识更新更稳定。
MEMIT-Merge: Addressing MEMIT's Key-Value Conflicts in Same-Subject Batch Editing for LLMs
- 同主题多条知识批量编辑时，合并值计算过程避免键值冲突。
- 在大批次下编辑成功率从50%提升至90%以上。
- 适合需要高频、多条同主题知识更新的场景。
随着大语言模型规模扩大，无需全量重训练即可修改模型内部知识的技术受到广泛关注。MEMIT作为一种主流的批量编辑算法，具备大规模知识修改能力。然而我们发现，当批量处理包含相同主题多条编辑时，MEMIT的编辑效果显著下降。分析表明，这源于其键值建模框架：相同的键（由共享主题生成）被强制表示不同的值（对应不同知识），导致编辑更新时产生冲突。为此，我们提出MEMIT-Merge，通过合并同主题事实的值计算过程，有效解决同主题批量编辑中的性能退化问题。实验结果表明，当MEMIT在大批次下的编辑成功率降至约50%时，MEMIT-Merge仍保持超过90%的成功率，展现出对主题实体冲突的卓越鲁棒性。代码已开源。
原文摘要 · Abstract (English)
As large language models continue to scale up, knowledge editing techniques that modify models' internal knowledge without full retraining have gained significant attention. MEMIT, a prominent batch editing algorithm, stands out for its capability to perform mass knowledge modifications. However, we uncover that MEMIT's editing efficacy significantly deteriorates when processing batches containing multiple edits sharing the same subject. Our analysis reveals this stems from MEMIT's key value modeling framework: identical keys (derived from the shared subject) are forced to represent different values (corresponding to different knowledge), resulting in update conflicts during editing. Addressing this issue, we propose MEMIT-Merge, an enhanced approach that merges value computation processes for facts sharing the same subject, effectively resolving the performance degradation in samesubject batch editing scenarios. Experimental results demonstrate that when MEMIT's edit success rate drops to around 50% at larger batch sizes, MEMIT-Merge maintains a success rate exceeding 90%, showcasing remarkable robustness to subject entity collisions. The code is available at https://github.com/NUSTM/ MEMIT-Merge.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。