通过定位梯度提升语言模型持续预训练效果
Gradient Localization Improves Lifelong Pretraining of Language Models
- 根据梯度大小识别知识所在层，针对性更新参数
- 在含时间漂移的文本上持续训练，性能提升12.3%
- 适合需要长期学习新知识的模型开发者
大规模语言模型（LLMs）在海量网络文本上训练后，其参数中已蕴含世界知识。然而，语言模型如何存储不同类型的知识仍不明确。本文研究了与时间敏感实体相关的两类知识，发现它们分别集中于模型不同层的参数中。我们推测，现有持续学习方法未考虑知识的局部性，是导致新知识无法有效吸收、旧知识发生灾难性遗忘的原因。实验观察到，包含更新或新提及实体的序列，在部分层中产生更大的梯度范数。通过将参数更新聚焦于这些相关层，显著提升了在存在时间漂移的语言数据上的持续预训练性能。
原文摘要 · Abstract (English)
Large Language Models (LLMs) trained on web-scale text corpora have been shown to capture world knowledge in their parameters. However, the mechanism by which language models store different types of knowledge is poorly understood. In this work, we examine two types of knowledge relating to temporally sensitive entities and demonstrate that each type is localized to different sets of parameters within the LLMs. We hypothesize that the lack of consideration of the locality of knowledge in existing continual learning methods contributes to both: the failed uptake of new information, and catastrophic forgetting of previously learned information. We observe that sequences containing references to updated and newly mentioned entities exhibit larger gradient norms in a subset of layers. We demonstrate that targeting parameter updates to these relevant layers can improve the performance of continually pretraining on language containing temporal drift.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。