arXiv:2609.05435cs.LGcs.CL2026-09

测试智能体能否从过往经验中持续学习,揭示模型真实成长能力。

AhaBench: Do Agents Learn from Prior Experience? A Benchmark for Long-Horizon Continual Learning

论文配图:AhaBench: Do Agents Learn from Prior Experience? A Benchmark for Long-Horizon Continual Learning
图 1 · 摘自论文原文
  • 设计三类任务检验长期连续学习:谜题探索、数学教学迁移、自动售货机运营。
  • 发现高初始表现的模型未必能持续进步,部分模型学习提升达25.8分。
  • 适合评估具长期适应能力的智能体,尤其关注延迟反馈与复杂场景应对。

现代语言智能体需在长时程中持续运作:提出追问、复用解题范例、处理工具反馈并适应延迟后果。现有评估多在单轮后重置或仅记录最终状态。AhaBench提出更实际的问题:固定模型在获得有效经验后,在支持条件被移除、改变或延迟的情况下,行为是否改善?该基准包含三个模块:Aha-Puzzle 测试解决隐藏状态谜题后的无提示探索能力;Aha-Euler 将 Project-Euler 风格数学问题转化为生成式教学/未见任务,配有精确验证器;Aha-Vending 是开源实现,受 Vending-Bench 启发,测试模拟售货机在延迟反馈与操作故障下是否仍盈利。基准报告三部分得分:初始分(Initial Score)、经验后得分(Post-Experience Score)和学习提升(Learning Lift)。实验显示,高初始表现、高后期表现与高学习提升模型不一致。在八模型对比中,Claude Opus 4.6 获得最高后验得分64.3与学习提升+25.8,Gemini 3.1 Pro 为63.4。组件分析表明:谜题任务虽提升支持下表现,但难转为无提示探索;数学任务全教学可达78.6%-100%,仅答案转移则在0.0%-73.9%;售货机任务区分了盈利处理与破产/无订单失败。已公开任务、评分标准、验证器、仿真代码与接口用于新模型评估。

原文摘要 · Abstract (English)

Modern language agents are expected to operate over long horizons: they ask follow-up questions, reuse worked examples, handle tool feedback, and adapt to delayed consequences. Most evaluations still reset the agent after a prompt or score only the final state of one trajectory. AhaBench asks a more operational question: when a fixed model receives useful experience, does its later behavior improve under a related evaluation condition where the obvious support has been removed, changed, or delayed? The suite contains three components. Aha-Puzzle tests no-hint exploration after solved hidden-state puzzles; Aha-Euler turns Project-Euler-style mathematical ideas into generated taught/held-out tasks with exact validators; and Aha-Vending, an open-source implementation inspired by Vending-Bench, tests whether a simulated vending agent remains profitable while handling delayed feedback and operational incidents. AhaBench reports a three-part scorecard: Initial Score measures starting competence, Post-Experience Score measures the later empirical outcome, and Learning Lift is their difference. This decomposition is the main empirical message: models that use visible support well, models that reach high post-experience scores, and models that improve most during a run are not always the same. On the common eight-model panel, Claude Opus 4.6 leads aggregate Post-Experience Score at 64.3 and aggregate Learning Lift at +25.8, with Gemini 3.1 Pro close behind at 63.4. The component results explain the split: puzzle traces raise supported scores but often fail to become no-hint exploration behavior; Aha-Euler full teaching reaches 78.6-100.0% while answer-only transfer ranges from 0.0 to 73.9%; and Aha-Vending separates profitable incident handling from bankruptcy and no-order failure. We release benchmark tasks, rubrics, validators, simulator code, and interfaces for evaluating new agents.

智能体持续学习评估基准长周期

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。