让大模型自己优化运行框架,提升任务成功率。
Self-Harness: Harnesses That Improve Themselves

- 大模型通过三阶段循环自动发现并修复自身运行缺陷。
- 在多个基准测试中,任务通过率最高提升132%。
- 适合想减少人工干预、自进化AI系统的开发者。
大模型代理的性能由基础模型和中介其与环境交互的框架共同决定。由于不同模型行为各异,有效框架设计本质上是模型相关的。然而,当前框架仍主要依赖人工专家设计,难以适应现代大模型日益多样化和快速演进的趋势。本文提出Self-Harness,一种大模型代理自我优化运行框架的新范式,无需人类工程师或更强外部代理。该范式通过三个阶段实现:弱点挖掘,从执行轨迹中识别模型特异性失败模式;框架提议,生成与这些失败关联的多样化且最小化的修改方案;提案验证,仅在回归测试通过后接受候选修改。我们在Terminal-Bench-2.0、SWE-bench Verified和AppWorld三个基准上,使用三个来自不同家族的基础模型(MiniMax M2.5、Qwen3.5-35B-A3B、GLM-5)及一个最小初始框架进行实验。在所有九种模型-基准组合中,最终框架均提升了保留内与保留外的通过率,相对提升最高达132%。定性分析表明,保留机制有效解决了产物处理、运行时控制、软件补丁验证及应用状态检索等基准特定瓶颈。结果表明,大模型不仅能被框架塑造,还能参与重塑自身框架。
原文摘要 · Abstract (English)
The performance of LLM-based agents is jointly shaped by their base models and the harnesses that mediate their interaction with the environment. Because different models exhibit distinct behaviors, effective harness design is inherently model-specific. Yet agent harnesses are still largely engineered by human experts, a paradigm that scales poorly as modern LLMs become increasingly diverse and rapidly evolving. In this paper, we introduce Self-Harness, a new paradigm in which an LLM-based agent improves its own operating harness, without relying on human engineers or stronger external agents. We operationalize Self-Harness as an iterative loop with three stages: Weakness Mining, which identifies model-specific failure patterns from execution traces; Harness Proposal, which generates diverse yet minimal harness modifications tied to these failures; and Proposal Validation, which accepts candidate edits only after regression testing. We instantiate Self-Harness across Terminal-Bench-2.0, SWE-bench Verified, and AppWorld using a minimal initial harness and three base models from diverse families: MiniMax M2.5, Qwen3.5-35B-A3B, and GLM-5. Across all nine model--benchmark combinations, every final harness improves both held-in and held-out pass rates, with overall relative gains of up to 132%. Qualitative analyses further show that the retained mechanisms address benchmark-specific bottlenecks in artifact handling and runtime control, software-patch verification, and application-state retrieval. These results suggest a path toward LLM-based agents that are not merely shaped by their harnesses, but can also participate in reshaping them.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。