测试智能体优化能否持续累积,发现只有带回归控制的方法能长期提升性能。
Do Agent Optimizers Compound? A Continual-Learning Evaluation on Terminal-Bench 2.0

- 在终端任务上分两阶段进行持续学习评估,检验优化效果是否可叠加。
- 仅RELAI-VCL方法在新增任务下仍能持续提升,最终通过率高达76.4%。
- 优化中加入回归控制是实现性能累积的关键,防止过拟合短视解。
当前多数智能体优化方法的收益为一次性:在固定基准上优化后报告提升,但未检验部署场景下的递归优化需求。本文提出核心问题:优化收益能否持续叠加?即在首次优化后,面对新任务能否再次优化而不损失已有成果?基于Terminal-Bench 2.0中的高难度任务,构建两阶段持续学习评估,对比三种方法(GEPA、Meta Harness、RELAI-VCL)在相同优化预算下的表现。三者在静态单阶段设置中均优于基线。但引入新任务后,差异显著:GEPA优化后迁移低于基线,Meta Harness迁移良好但无法二次提升,而RELAI-VCL唯一实现正向迁移且持续改进,各阶段通过率最高,终身平均通过率达76.4%(GEPA 66.0%,Meta Harness 64.6%,基线58.7%)。关键发现:仅当优化循环内嵌回归控制时,优化收益才能真正累积,形成对泛化失效的归纳偏置。
原文摘要 · Abstract (English)
Most reported gains from agent-optimization methods are one-shot: an agent is optimized against a fixed benchmark and the resulting improvement is reported as if it were a stable property of the method. This does not test the setting that matters for deployed agents, where optimization is applied recursively as new failures and new tasks appear over time. The central question this raises is whether optimizer-driven gains compound: after an agent has been optimized once, can it be optimized again on newly arrived tasks without eroding the gains the first round produced? We study this question with a two-phase continual-learning evaluation built from hard tasks in Terminal-Bench 2.0, comparing three approaches to agent-harness optimization (GEPA, Meta Harness, and RELAI's Verifiable Continual Learning, RELAI-VCL) under identical optimization budgets. All three methods improve over the baseline agent in the conventional, static, single-phase setting. However, once new tasks are introduced, the methods diverge sharply: GEPA's optimized agent transfers below the unoptimized baseline, Meta Harness transfers well but fails to improve further once given a second optimization budget, and RELAI-VCL is the only method that both transfers positively to unseen tasks and continues improving after those tasks are folded into the optimization objective, reaching the highest pass rate at every evaluated stage and the highest lifelong average pass rate overall (76.4% vs. 66.0% for GEPA, 64.6% for Meta Harness, and 58.7% for the baseline). Our key observation was that optimization gains compounded only when regression control was built into the optimization loop, providing an inductive bias against shortcut solutions that fail to generalize.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。