发现大模型在多次知识更新后检索偏差加剧，越早的知识越易被正确召回。
Diagnosing Retrieval Bias Under Multiple In-Context Knowledge Updates in Large Language Models
- 模拟多次知识更新，用早期和最新状态对比评估模型检索能力。
- 随着更新次数增加，最新状态准确率显著下降，早期状态仍保持高位。
- 注意力与隐藏状态信号变平，难以区分新旧知识，适合关注长上下文推理的团队参考。
大型语言模型广泛应用于需依赖知识的任务中，同一事实可能在上下文中被多次修订。现有研究多聚焦单次更新或单一冲突，而多轮更新场景下，多个历史有效版本会在检索时竞争，但尚未得到充分探讨。该问题类似于认知心理学中的AB-AC干扰范式：当同一线索A先后关联B和C时，新旧关联在回忆阶段产生竞争，导致偏差。受此启发，我们提出动态知识实例（DKI）评估框架，将同一事实的多次更新建模为线索与一系列更新值的配对，通过端点探针检测初始状态与最新状态的表现。在多种LLM上，我们发现随着更新次数增加，检索偏差加剧；初始状态准确率保持稳定，而最新状态准确率大幅下降。对注意力、隐藏状态相似性和输出逻辑值的诊断分析显示，错误情况下这些信号趋于平坦且缺乏区分性，难以提供稳定依据以识别最新更新。最后，受认知启发的启发式干预策略仅带来轻微提升，无法消除偏差。结果揭示了在长上下文中追踪和跟随知识更新的持续挑战。
原文摘要 · Abstract (English)
LLMs are widely used in knowledge-intensive tasks where the same fact may be revised multiple times within context. Unlike prior work focusing on one-shot updates or single conflicts, multi-update scenarios contain multiple historically valid versions that compete at retrieval, yet remain underexplored. This challenge resembles the AB-AC interference paradigm in cognitive psychology: when the same cue A is successively associated with B and C, the old and new associations compete during retrieval, leading to bias. Inspired by this, we introduce a Dynamic Knowledge Instance (DKI) evaluation framework, modeling multi-updates of the same fact as a cue paired with a sequence of updated values, and assess models via endpoint probing of the earliest (initial) and latest (current) states. Across diverse LLMs, we observe that retrieval bias intensifies as updates increase, earliest-state accuracy stays high while latest-state accuracy drops substantially. Diagnostic analyses of attention, hidden-state similarity, and output logits further reveal that these signals become flatter and weakly discriminative on errors, providing little stable basis for identifying the latest update. Finally, cognitively inspired heuristic intervention strategies yield only modest gains and do not eliminate the bias. Our results reveal a persistent challenge in tracking and following knowledge updates in long contexts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。