测试大模型对持续更新知识流的实时适应能力
Can Large Language Models Keep Up? Benchmarking Online Adaptation to Continual Knowledge Streams
- 构建动态知识流基准OAKS，模拟事实随时间反复变化
- 14个模型均出现状态追踪延迟和注意力分散问题
- 适合研究在线学习、持续推理与记忆机制的学者
在动态真实场景中运行的大语言模型常面临持续演化或渐进涌现的知识。为保持准确性和有效性，模型必须能够实时适应新信息。本文提出在线适应持续知识流（OAKS）基准，用于评估模型对不断更新知识流的适应能力。该基准由一系列细粒度上下文片段构成，其中事实在时间区间内动态变化。OAKS包含两个数据集：OAKS-BABI和OAKS-Novel，每个数据集中个体事实在多个上下文片段中多次演变，并配有密集标注以衡量模型对变化的追踪准确性。对14种具有不同推理方式的模型进行评估，发现当前方法存在显著局限：最先进模型与代理记忆系统均无法在OAKS上稳健适应，表现出状态追踪延迟和在流式环境中易受干扰的问题。
原文摘要 · Abstract (English)
LLMs operating in dynamic real-world contexts often encounter knowledge that evolves continuously or emerges incrementally. To remain accurate and effective, models must adapt to newly arriving information on the fly. We introduce Online Adaptation to Continual Knowledge Streams(OAKS) to evaluate this capability, establishing a benchmark for online adaptation over streaming, continually updating knowledge. Specifically, the benchmark is structured as a sequence of fine-grained context chunks where facts change dynamically across time intervals. OAKS comprises two datasets: OAKS-BABI and OAKS-Novel, where individual facts evolve multiple times across context chunks. These datasets include dense annotations to measure whether models track changes accurately. Evaluating 14 models with varied inference approaches, we observe significant limitations in current methodologies. Both state-of-the-art models and agentic memory systems fail to adapt robustly on OAKS, demonstrating delays in state-tracking and susceptibility to distraction within streaming environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。