自动演化代码代理的测试框架,让系统自我改进。
Agentic Harness Engineering: Observability-Driven Automatic Evolution of Coding-Agent Harnesses

- 构建可观察的闭环系统,让每个修改都有据可查
- 10轮迭代后任务通过率从69.7%提升至77.0%
- 进化出的组件具通用性,跨模型家族表现更优
测试框架现已成为影响代码代理性能的核心因素,决定了模型如何与工具和执行环境交互。然而,框架设计仍依赖人工,因可编辑组件异构、轨迹数据庞大且信号被淹没、修改效果难以归因。本文提出自驱动测试框架工程(AHE),通过三大可观测支柱实现自动化:(1) 组件可观测性为每个可编辑部分提供文件级表征,使操作空间明确且可回滚;(2) 经验可观测性将数百万原始轨迹令牌压缩为分层可钻取的证据库,供演化代理实际使用;(3) 决策可观测性将每次修改与其自声明预测配对,后续任务结果验证其有效性。三者结合使每次修改成为可验证契约,实现自主演化而非盲目试错。实证显示,10轮AHE迭代使Terminal-Bench 2的pass@1从69.7%提升至77.0%,超过人工设计框架Codex-CLI(71.9%)及自演化基线ACE与TF-GRPO。冻结后的框架无需重演:在SWE-bench-verified上以少12%的token达成最高综合成功率;在Terminal-Bench 2上对三个不同模型族分别获得+5.1至+10.1个百分点的提升,表明进化组件编码的是通用工程经验而非特定基准调优。消融实验显示收益主要来自工具、中间件与长期记忆,而非系统提示词,说明结构层面可迁移,而策略文本不可迁移。
原文摘要 · Abstract (English)
Harnesses are now central to coding-agent performance, mediating how models interact with tools and execution environments. Yet harness engineering remains a manual craft, because automating it faces a heterogeneous action space across editable components, voluminous trajectories that bury actionable signal, and edits whose effect is hard to attribute. We introduce Agentic Harness Engineering (AHE), a closed loop that addresses these challenges through three matched observability pillars: (1) component observability gives every editable harness component a file-level representation so the action space is explicit and revertible; (2) experience observability distills millions of raw trajectory tokens into a layered, drill-down evidence corpus that an evolving agent can actually consume; and (3) decision observability pairs every edit with a self-declared prediction, later verified against the next round's task-level outcomes. Together, these pillars turn every edit into a falsifiable contract, so harness evolution proceeds autonomously without collapsing into trial-and-error. Empirically, ten AHE iterations lift pass@1 on Terminal-Bench 2 from 69.7% to 77.0%, surpassing the human-designed harness Codex-CLI (71.9%) and the self-evolving baselines ACE and TF-GRPO. The frozen harness transfers without re-evolution: on SWE-bench-verified it tops aggregate success at 12% fewer tokens than the seed, and on Terminal-Bench 2 it yields +5.1 to +10.1pp cross-family gains across three alternate model families, indicating the evolved components encode general engineering experience rather than benchmark-specific tuning. Ablations localize the gain to tools, middleware, and long-term memory rather than the system prompt, suggesting factual harness structure transfers while prose-level strategy does not.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。