arXiv:2606.05661cs.AIcs.CL2026-06被引 5

首个跨领域专家验证的持续学习基准,测试大模型能否真正从经验中进步。

Continual Learning Bench: Evaluating Frontier AI Systems in Real-World Stateful Environments

论文配图:Continual Learning Bench: Evaluating Frontier AI Systems in Real-World Stateful Environments
图 1 · 摘自论文原文
  • 构建六领域专家验证任务,设计可在线发现的潜在结构
  • 发现主流模型普遍过拟合或无法复用知识,记忆系统未改善表现
  • 适合评估智能体持续学习能力,推动更鲁棒的系统设计

持续学习指AI系统通过连续经验不断改进,但目前缺乏高质量评估基准。我们提出首个困难且经专家验证的基准CL-Bench,用于衡量基于大语言模型的系统是否真正实现经验积累。该基准覆盖六个不同领域:软件工程、信号处理、疾病暴发预测、数据库查询、策略游戏和需求预测,每个任务均由领域专家验证,并设计为包含可学习的潜在结构(如代码库布局、疫情动态、对手策略),只有具备状态记忆的系统才能在线发现,而无状态系统无法做到。我们在多种代理架构上评估前沿模型,包括简单的上下文学习(ICL)到专用记忆系统,引入增益指标以分离学习效应与原有能力。结果表明,现有系统仍有巨大提升空间:代理常对即时观测过拟合,或无法跨实例复用知识;专用记忆系统并未解决此问题,甚至朴素的ICL表现优于专门的记忆管理方案。CL-Bench是首个在多样化真实世界领域中,结合专家验证任务并分离在线学习与模型固有能力的基准,揭示了对更优持续学习系统的需求。

原文摘要 · Abstract (English)

Continual learning, the ability of AI systems to improve through sequential experience, has attracted substantial interest, but no high-quality benchmark exists to evaluate it. We introduce Continual Learning Bench (CL-Bench), the first difficult, expert-validated benchmark designed to measure whether LLM-based systems genuinely improve with experience. CL-Bench spans six diverse domains (software engineering, signal processing, disease outbreak forecasting, database querying, strategic game-playing, and demand forecasting), each validated by domain experts and designed so that tasks share a learnable latent structure (codebase layout, disease outbreak dynamics, opponent strategies) that a stateful system can discover online but a stateless one cannot. We evaluate frontier models across several agent architectures, from naive in-context learning (ICL) to dedicated memory systems, introducing a gain metric to isolate learning from prior capabilities. We find that these systems leave headroom for improved continual learning: agents frequently overfit to immediate observations or fail to reuse knowledge across instances, and dedicated memory systems do not fix this -- in fact, naive ICL outperforms systems dedicated to memory management. CL-Bench is the first benchmark to evaluate continual learning across diverse real-world domains with expert-validated tasks and isolate online learning from underlying model capability, showing a need for better continual learning systems.

持续学习大模型评估基准测试智能体

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。