编码代理评估中，支架选择比模型本身影响更大，决定效率与成本。
The Scaffold Effect in Coding Agents: Harness Choice as a Hidden Variable in Coding-Agent Evaluation
- 用三个开源支架对比同一模型性能，发现支架差异导致每解题耗能差40倍。
- 同一模型在不同支架下通过率仅差0-8个百分点，说明效率差异主要来自支架。
- 建议按令牌/延迟预算选支架-模型组合，并公开完整配置和日志。
公开排行榜通常以模型名称和通过率排名，但支撑系统（即支架）的定义常不明确。当支架固定时，模型间比较有效；若支架变化，则性能与效率混杂了模型与支架的影响。我们在终端基准测试专业版的50个分层任务子集上，对Qwen 3.6 Plus和MiniMax M2.5在三个开源支架（Goose、OpenCode、OpenHands-SDK）下进行评估。支架选择导致每解题消耗的令牌数相差最高达40倍，而同一模型内通过率差异仅0-8个百分点（95%置信区间包含零，除最大差距外）。失败模式在不同模型间重复出现（如Goose为REASON，OpenHands-SDK为VERIFY/MAX_TURNS，OpenCode为idle-loop/TIME），表明支架层面存在模型无关的偏差。因此，对于以人为中心的编码代理评估，仅凭模型名称不足以衡量表现；支架-模型组合决定了真实世界中的成本、延迟和监督负担；无操作回合是任务级等待税，不仅是令牌税。我们建议根据在令牌/延迟预算下的通过率选择支架-模型组合，并在报告模型比较时同步提供令牌使用量、延迟及完整支架配置。代码、原始日志、聚合快照和分析脚本已公开。
原文摘要 · Abstract (English)
Public leaderboards for coding agents typically rank systems by model name and pass rate, while the surrounding harness (the scaffold that issues tools, manages context, and decides when to stop) is often under-specified. Model-to-model comparison is valid when the harness is fixed; when it varies, performance and efficiency conflate model and scaffold effects. We evaluate Qwen 3.6 Plus and MiniMax M2.5 across three open-source harnesses (Goose, OpenCode, OpenHands-SDK) on a stratified 50-task subset of Terminal-Bench Pro. Harness choice induces up to a 40x difference in tokens per solved task, while paired within-model pass-rate differences remain 0-8 percentage points (95% paired-task bootstrap CIs include zero except for the largest gap). Failure fingerprints replicate across models (REASON for Goose, VERIFY/MAX_TURNS for OpenHands-SDK, idle-loop/TIME for OpenCode), indicating harness-level biases that are largely model-independent. For human-centered coding-agent evaluation, model name alone is an incomplete comparison unit: harness-model pairs determine real-world cost, latency, and oversight burden; no-action turns are a per-task wait tax, not just a token tax. We therefore recommend selecting harness-model pairs by pass rate under token/latency budgets, and reporting token usage, latency, and full harness specifications alongside any model comparison. We release anonymized configs, raw trial logs, aggregated snapshots, and analysis scripts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。