自动优化LLM代理的外部框架,提升长任务可靠性
AutoSaddler: Automatic Harness Optimization with Durable Updates from Agent Execution Traces

- 从执行失败轨迹中诊断问题,像改代码一样生成修复方案
- 在三个基准上分别提升9.0、9.6、10.0个百分点
- 适合想提升复杂任务鲁棒性的研究者与工程师
LLM代理在长周期任务中仍不可靠,微小局部失败会随交互累积导致整体失败。尽管外部框架能显著提升鲁棒性,但其设计需手动搜索大量提示、工具配置和控制逻辑,过程耗时昂贵。我们提出AutoSaddler,一个自动框架,将框架优化建模为离线学习问题,通过小批量失败信号迭代更新。该方法结合失败轨迹诊断、将框架视为代码的结构化补丁生成,以及基于验证的更新选择。在GAIA2、SWE-Bench Pro和Terminal-Bench 2.0上的实验表明,AutoSaddler相比基线框架分别提升9.0、9.6和10.0个百分点。消融实验显示:深度调试优于浅层反思,针对性修改优于无约束编辑,泛化感知选择优于仅针对特定轨迹修复。这些结果表明,自动框架优化是提升代理系统性能与可靠性的可行路径。
原文摘要 · Abstract (English)
LLM agents remain unreliable on long-horizon tasks, where small local failures can compound over extended interactions and lead to overall task failure. Although external harnesses can substantially improve robustness, harness design remains a manual and expensive process that requires searching over a large space of prompts, tool configurations, and control logic. We propose AutoSaddler, an automatic harness optimization framework that formulates harness improvement as an offline learning problem and iteratively updates the harness using failure signals from mini-batches. AutoSaddler combines failure-trace diagnosis, structured patch generation that treats the harness as code, and validation-based update selection. Experiments on GAIA2, SWE-Bench Pro, and Terminal-Bench 2.0 show that AutoSaddler substantially improves agent performance over the corresponding base harnesses, achieving gains of 9.0, 9.6, and 10.0 percentage points, respectively. Ablation studies further suggest that effective harness optimization benefits from three ingredients: deep debugging rather than shallow reflection, targeted modifications rather than unconstrained editing, and generalization-aware selection rather than trajectory-specific repair. Together, these results suggest that automatic harness optimization is a promising path toward more performant and reliable agent systems.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。