arXiv:2506.04226cs.CLcs.AI2025-06ACL被引 4

只需预计算极少量隐藏向量,就能高效更新知识,大幅缩短准备时间。

Efficient Knowledge Editing via Minimal Precomputation

  • 只预计算原方案0.3%的隐藏向量,仍能实现知识编辑
  • 将预计算时间从数十小时缩短至数分钟
  • 适用于需快速部署知识更新的模型应用

知识编辑方法如MEMIT可通过单句高效更新事实及其后果,但其背后存在显著的预计算步骤:原始方法需对每层预计算约4400万隐藏向量,相当于对4400万个标记进行前向传播。对于GPT-J(6B)和Llama2-7B,该步骤分别耗时约36小时和40小时,且随模型规模增长。本文证明此开销可大幅削减:理论上最小预计算量已知,实验证明仅需少于0.3%的原定隐藏向量即可完成编辑。该方法使用户可在几分钟内开始编辑新模型,极大提升效率。

原文摘要 · Abstract (English)

Knowledge editing methods like MEMIT are able to make data and compute efficient updates of factual knowledge by using a single sentence to update facts and their consequences. However, what is often overlooked is a "precomputation step", which requires a one-time but significant computational cost. The authors of MEMIT originally precompute approximately 44 million hidden vectors per edited layer, which requires a forward pass over 44 million tokens. For GPT-J (6B), this precomputation step takes 36 hours on a single GPU, while it takes approximately 40 hours for Llama2-7B. Additionally, this precomputation time grows with model size. In this paper, we show that this excessive computational cost is unnecessary. Knowledge editing using MEMIT and related methods, such as ROME and EMMET, can be performed by pre-computing a very small portion of the 44 million hidden vectors. We first present the theoretical minimum number of hidden vector precomputation required for solutions of these editing methods to exist. We then empirically show that knowledge editing using these methods can be done by pre-computing significantly fewer hidden vectors. Specifically, we show that the precomputation step can be done with less than 0.3% of the originally stipulated number of hidden vectors. This saves a significant amount of precomputation time and allows users to begin editing new models within a few minutes.

知识编辑模型优化预计算

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。