提出MEMAT方法，用注意力机制高效跨语言更新大模型知识
Mass-Editing Memory with Attention in Transformers: A cross-lingual exploration of knowledge
- 通过注意力机制实现大规模知识编辑，仅需少量参数改动
- 在所有指标上提升10%，跨语言泛化能力显著增强
- 适合需要多语言知识更新的模型维护场景
近期研究探索了在大型语言模型中更新和修改事实知识的方法，通常聚焦于特定的多层感知机模块。本研究拓展了这一方向，考察现有知识编辑方法在不同语言中的有效性，并深入探讨注意力机制在此过程中的作用。基于所得洞察，我们提出一种名为Mass-Editing Memory with Attention in Transformers（MEMAT）的方法，在几乎不改变参数的前提下，实现了各项指标的显著提升。MEMAT在度量值上带来高达10%的提升，不仅对训练数据中未包含的语言有效，还表现出极强的可迁移性。代码与数据已公开于https://github.com/dtamayo-nlp/MEMAT。
原文摘要 · Abstract (English)
Recent research has explored methods for updating and modifying factual knowledge in large language models, often focusing on specific multi-layer perceptron blocks. This study expands on this work by examining the effectiveness of existing knowledge editing methods across languages and delving into the role of attention mechanisms in this process. Drawing from the insights gained, we propose Mass-Editing Memory with Attention in Transformers (MEMAT), a method that achieves significant improvements in all metrics while requiring minimal parameter modifications. MEMAT delivers a remarkable 10% increase in magnitude metrics, benefits languages not included in the training data and also demonstrates a high degree of portability. Our code and data are at https://github.com/dtamayo-nlp/MEMAT.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。