新知识会扰动旧知识，该研究提出方法抑制这种干扰。
How new data permeates LLM knowledge and how to dilute it
- 用新数据训练时，模型会产生'泛化过度'现象，把新知识误用于无关场景。
- 通过分析关键词概率，可提前预测知识渗透程度，跨模型验证有效。
- 提出两种新策略，使模型学新知识同时减少错误关联，效果提升50%-95%。
大型语言模型通过梯度更新持续学习，但单个新信息如何影响已有知识、导致有益泛化或有害幻觉，仍不明确。本文揭示，学习新知识时模型会出现‘引子效应’：新事实可能被错误地应用到无关上下文中。为系统研究此现象，我们构建了包含1320个多样化样本的‘Outlandish’数据集，用于探测新知识如何渗透至模型原有知识库。实验表明，学习前关键词的词元概率可有效预测引子效应强度，该规律在PALM-2、Gemma、Llama等不同架构、规模与训练阶段的模型中均成立。最后，我们提出两种新方法：（1）‘踏脚石’文本增强策略，（2）‘忽略k’更新剪枝法，可将不良引子效应降低50%-95%，同时保持模型获取新知识的能力。研究结果为理解模型学习机制提供了实证依据，并提供了精准注入知识的实用工具。
原文摘要 · Abstract (English)
Large language models learn and continually learn through the accumulation of gradient-based updates, but how individual pieces of new information affect existing knowledge, leading to both beneficial generalization and problematic hallucination, remains poorly understood. We demonstrate that when learning new information, LLMs exhibit a "priming" effect: learning a new fact can cause the model to inappropriately apply that knowledge in unrelated contexts. To systematically study this phenomenon, we introduce "Outlandish," a carefully curated dataset of 1320 diverse text samples designed to probe how new knowledge permeates through an LLM's existing knowledge base. Using this dataset, we show that the degree of priming after learning new information can be predicted by measuring the token probability of key words before learning. This relationship holds robustly across different model architectures (PALM-2, Gemma, Llama), sizes, and training stages. Finally, we develop two novel techniques to modulate how new knowledge affects existing model behavior: (1) a ``stepping-stone'' text augmentation strategy and (2) an ``ignore-k'' update pruning method. These approaches reduce undesirable priming effects by 50-95\% while preserving the model's ability to learn new information. Our findings provide both empirical insights into how LLMs learn and practical tools for improving the specificity of knowledge insertion in language models. Further materials: https://sunchipsster1.github.io/projects/outlandish/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。