arXiv:2409.18679cs.CL2024-09被引 1

不同问题类型对模型编辑副作用影响差异大,可优化知识更新实验设计。

"Why" Has the Least Side Effect on Model Editing

  • 按问题类型分类分析模型编辑副作用,揭示其影响差异
  • 问题类型不同导致性能下降幅度差异显著,最大差值达30%以上
  • 小模型结论不适用于大模型,且大批次训练可减轻副作用

从头训练大型语言模型(LLMs)成本高昂,尤其在世界知识持续演进的背景下。为保持模型相关性与准确性,模型编辑成为关键研究方向。尽管该方法前景广阔,但常引发意外副作用,其成因尚不明确。本文通过将模型编辑问题按类型分类,发现不同问题类型导致的性能退化程度差异显著,为知识编辑的实验设计提供了新见解。同时,我们验证了小模型的发现能否推广至大模型,结果表明两者存在显著差异,提示小模型结论未必适用于大模型。此外,研究发现增大批大小可有效缓解性能下降。

原文摘要 · Abstract (English)

Training large language models (LLMs) from scratch is an expensive endeavor, particularly as world knowledge continually evolves. To maintain relevance and accuracy of LLMs, model editing has emerged as a pivotal research area. While these methods hold promise, they can also produce unintended side effects. Their underlying factors and causes remain largely unexplored. This paper delves into a critical factor-question type-by categorizing model editing questions. Our findings reveal that the extent of performance degradation varies significantly across different question types, providing new insights for experimental design in knowledge editing. Furthermore, we investigate whether insights from smaller models can be extrapolated to larger models. Our results indicate discrepancies in findings between models of different sizes, suggesting that insights from smaller models may not necessarily apply to larger models. Additionally, we examine the impact of batch size on side effects, discovering that increasing the batch size can mitigate performance drops.

模型编辑大模型副作用实验设计

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。