arXiv:2608.06301cs.AIcs.CL2026-08被引 1

评测大模型自动优化任务调度代码的能力,发现性能差异显著。

HarnessOpt-Bench: Evaluating LLMs at Harness Optimization

论文配图:HarnessOpt-Bench: Evaluating LLMs at Harness Optimization
图 1 · 摘自论文原文
  • 用大模型自动改写任务调度代码,通过反馈不断优化
  • 在111次实验中,不同模型提升幅度差异大,最高超基线37%
  • 适合研究智能体系统、自动化调试与提示工程的开发者

随着大模型越来越多地应用于智能体系统,其表现不仅取决于模型权重,还依赖于‘调度环境’:包括提示词、工具、控制流、记忆和编排代码。因此,自动化调度优化——即由AI系统通过迭代与评估引导改进调度环境——既是提升AI系统的关键路径,也对AI自身能力提出高要求。但社区缺乏统一衡量前沿大模型在此任务上表现的标准。我们提出HarnessOpt-Bench,一个在昂贵且随机评估条件下进行端到端调度优化的基准。优化器(大模型+代码调度环境)接收目标智能体的初始调度配置、评分反馈和固定评估预算,通过编辑调度并提交最终候选方案;该方案在未公开的测试集上以归一化增益评分。可信执行环境确保评估边界、计量资源消耗并保存候选版本供审计。我们在4个下游任务上,对5个前沿大模型作为优化器,在共享编码调度环境和各自原生调度环境中进行了评估,共完成111次评分运行。结果表明,优化器之间的差异大于其所依赖的调度环境,原生调度环境并非始终更优,且增益在不同任务和初始设置间波动显著。这些结果确立了调度优化为可测量、具区分力的能力,仍有巨大提升空间。

原文摘要 · Abstract (English)

As LLMs are increasingly deployed within agentic systems, their capabilities depend not only on the model weights but also on the harness: the prompts, tools, control flow, memory, and orchestration code surrounding them. This makes automated harness optimization -- the iterative and evaluation-guided improvement of a harness by an AI system -- both an important route to improving AI systems and a demanding capability for AI systems themselves. Yet the community lacks a common protocol for measuring how well frontier LLMs perform at this task. We introduce HarnessOpt-Bench, a benchmark for end-to-end harness optimization under expensive and stochastic evaluation. An optimizer, an LLM paired with a coding harness, receives a target agent's seed harness, graded evaluation feedback, and a fixed target-evaluation budget. It edits the harness and nominates a final candidate, which is scored by its normalized gain over the seed on a held-out test partition that remains inaccessible throughout search. A trusted execution environment enforces the evaluation boundary, meters target-agent resource use, and preserves candidate versions for audit. We evaluate 5 frontier LLMs as optimizers both under a shared coding harness and under their native harnesses across 4 downstream tasks, over 111 scored runs. Experiment results show that optimizer models separate more than the coding harnesses they act through, native harnesses are not consistently superior, and gains vary substantially across tasks and seed regimes. These results establish harness optimization as a measurable and discriminative capability with large space for improvement.

大模型优化智能体系统自动化调试

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。