arXiv:2604.07236cs.AIcs.CL2026-04

拆解智能体的决策流程,发现大模型实际贡献有限。

How Much Heavy Lifting Can an Agent Harness Do?: Measuring the LLM's Residual Role in a Planning Agent

  • 将智能体分层为四层,逐层测量每部分对性能的贡献。
  • 显式规划层提升胜率24.1个百分点,无需调用大模型。
  • 大模型仅在4.3%回合被调用,且效果非单调递增。

Agent harnesses——即封装语言模型的状态化程序,决定每一步模型的输入——已被证实可使固定模型的端到端性能提升六倍。这引出一个被忽视的问题:智能体的能力有多少来自其自身,又有多少真正依赖大语言模型?本文将带有噪声的协作战舰游戏的规划智能体外部化为四个逐步增强的层级:后验信念跟踪、命题式规划、符号化反思和基于大模型的修订门,并在统一运行时下评估,以胜率(win rate)为主要指标,F1为次要指标,预先定义‘重载’为对主指标的最大正向边际贡献。在54局游戏中,命题式规划承担了主要作用(相比仅信念追踪的智能体提升24.1个百分点胜率,且不调用大模型);符号化反思机制有效但校准敏感,棋盘级影响最高达±0.140 F1,整体抵消;基于大模型的修订仅在4.3%回合激活,效果有界且非单调。研究贡献在于方法论:一旦各层级可外部度量,大模型的角色便能被量化为残余而非中心假设。

原文摘要 · Abstract (English)

Agent harnesses -- the stateful programs that wrap a language model and decide what it sees at each step -- are now known to change end-to-end performance on a fixed model by as much as six times. That raises a question asked less often than it should be: how much of an agent's competence does the harness itself already carry, and how much genuinely still needs the LLM? We externalize a planning harness for noisy Collaborative Battleship into four progressively richer layers -- posterior belief tracking, declarative planning, symbolic reflec tion, and an LLM-backed revision gate -- under a common runtime, taking \emph{win rate} as the primary metric and \emph{F1} as secondary, and pre-specifying \emph{heavy lifting} as the single largest positive marginal to the primary metric. Across 54 games, declarative pla nning carries the heavy lifting ($+24.1$pp win rate over a belief-only harness, zero LLM calls); symbolic reflection is mechanistically real but calibration-sensitive, with signed board-level effects up to $\pm0.140$ F1 that cancel on aggregate; and LLM-backed revision ac tivates on only $4.3\%$ of turns with a bounded, non-monotonic effect. The contribution is methodological: once harness layers are made externally measurable, the LLM's role can be quantified as residual rather than assumed central.

智能体大模型量化评估

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。