arXiv:2605.23950cs.AIcs.SE2026-05被引 10

评估大模型智能体时,执行框架比模型本身影响更大,当前排行榜可能误导结论。

Stop Comparing LLM Agents Without Disclosing the Harness

论文配图:Stop Comparing LLM Agents Without Disclosing the Harness
图 1 · 摘自论文原文
  • 将执行框架视为闭环控制系统的控制器,解释其对性能的决定性作用
  • 实验证明框架差异导致的性能波动远超模型替换的影响,甚至逆转排名
  • 提出需披露框架配置的评估标准,否则排行榜不完整且易误导

本文主张,在面向长周期任务、模型能力相近的场景下,智能体执行框架(即控制上下文构建、工具调用、编排与验证的基础设施层)对性能的影响往往超过其所封装的语言模型。我们提出并论证了‘绑定约束假说’:在此情境下,性能差异主要由框架配置决定,而非模型选择;现有评估协议因此系统性地将框架改进误归因于模型提升。支撑该假说的三方面证据包括:第一,基于控制理论的形式化将框架视为控制器,语言模型作为受控随机策略,解释为何微小框架调整可带来超越模型替换的性能变化;第二,公开基准测试、产业部署及受控方差分解均表明,框架引起的性能波动显著大于模型差异,甚至导致模型排名反转;第三,我们提出一个框架感知的评估框架,包含披露标准与方差分解协议。在框架规格未公开前,长周期智能体排行榜应被视为不完整且可能具有误导性。

原文摘要 · Abstract (English)

This position paper argues that, for long-horizon tasks evaluated across models with comparable frontier capability, the agent execution harness, namely the infrastructure layer that governs context construction, tool interaction, orchestration, and verification around a language model, is often a stronger determinant of agent performance than the model it wraps. We formalize and defend the Binding Constraint Thesis: in this regime, performance variance is governed more by harness configuration than by model choice, and current evaluation protocols therefore systematically misattribute harness-level gains to model improvements. We support this thesis along three lines. First, a control-theoretic formalization treats the harness as the controller of a closed-loop dynamical system and the LLM as the stochastic policy it governs, which explains why small harness changes can produce performance shifts that exceed those obtained by substituting one model for another. Second, published benchmarks, industry deployments, and a controlled variance decomposition show that harness-induced variance can substantially exceed model-induced variance, including cases of model ranking reversal. Third, we propose a harness-aware evaluation framework with a disclosure standard and a variance decomposition protocol. Until harness specifications are disclosed, leaderboard comparisons for long-horizon agents should be treated as incomplete and potentially misleading.

大模型智能体评估框架性能归因排行榜可信度

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。