提出新基准评估大模型应对持续知识演化的能力,发现现有方法普遍失效。
RAG or Learning? Understanding the Limits of LLM Adaptation under Continuous Knowledge Drift in the Real World
- 构建时间戳动态事件数据集,模拟真实世界知识演化过程。
- 多数方法(含RAG)在持续知识漂移下出现灾难性遗忘和时序不一致。
- 提出时序感知检索框架Chronos,无需训练即可提升推理一致性。
大语言模型的知识主要来自预训练阶段,难以适应持续演化的现实世界信息。随着事实、实体与事件随时间变化,模型易产生持续知识漂移,导致预测过时及推理时序矛盾。现有方法如持续微调、知识编辑和检索增强生成(RAG)虽旨在更新或补充知识,却极少在反映时间演进的真实场景中评估。本文构建了一个基于时间戳证据的实时动态事件基准,可系统评估模型在持续知识漂移下的适应能力。结果揭示,主流方法(包括基础RAG和多种学习型方法)均表现不佳,暴露出灾难性遗忘和时序不一致等关键缺陷。为此,我们提出时序感知检索基线Chronos,将检索到的证据逐步组织为事件演化图,实现无需额外训练的更连贯时序理解。本工作为真实场景下大模型应对持续知识漂移提供了分析与改进基础。
原文摘要 · Abstract (English)
Large language models (LLMs) acquire most of their knowledge during pretraining, which ties them to a fixed snapshot of the world and makes adaptation to continuously evolving knowledge challenging. As facts, entities, and events change over time, models may experience continuous knowledge drift, resulting not only in outdated predictions but also in temporally inconsistent reasoning. Although existing approaches, such as continual finetuning, knowledge editing, and retrieval-augmented generation (RAG), aim to update or supplement model knowledge, they are rarely evaluated in settings that reflect chronological, evolving, and real-world knowledge evolution. In this work, we introduce a new benchmark of real-world dynamic events, constructed from time-stamped evidence that captures how knowledge evolves over time, which enables systematic evaluation of model adaptation under continuous knowledge drift. The benchmark reveals that most existing methods, including vanilla RAG and several learning-based approaches, struggle under this setting, exposing critical limitations such as catastrophic forgetting and temporal inconsistency. To mitigate these limitations, we propose a time-aware retrieval baseline, Chronos, which progressively organizes retrieved evidence into an Event Evolution Graph to enable more temporally consistent understanding in LLMs without additional training. Overall, this work provides a foundation for analyzing and advancing LLM adaptation to continuous knowledge drift in realistic settings.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。