模型越强，越不一定需要复杂引导，且引导方式需按类型选择。
It's Not the Capability: Harness Sensitivity Is Non-Monotone Across LLM Agent Tiers
- 测试六种模型在三种引导策略下的表现，发现引导复杂度与效果非单调关系
- 顶尖聊天模型因引导过细，正确率下降29-38个百分点，出现反直觉现象
- 推理类模型在严格引导下表现最佳，适合不同模型类型定制引导策略
LLM代理部署中普遍假设：更结构化的引导能提升可靠性，且高能力模型需要更少引导，即模型能力与最优引导复杂度呈单调反比。我们通过432次受控实验，在包含24个任务的HEAT-24合成基准（基于git工作区验证）上，交叉测试了六种模型在四个能力层级下，三种引导条件（轻量、平衡、严格）的表现。结果表明该单调关系不成立：首先，对于前沿聊天模型Gemini 2.5 Flash，引导越冗长，VTSR降低29-38个百分点，形成引导复杂度悖论；其次，对于前沿推理模型Qwen3.5-122B（开启扩展思维），严格引导达到最高VTSR（91.7%）且延迟最低，与预测相反。在受限层级中，2B模型Gemma4:e2B在所有引导下均达91.7%稳定性。由于每层级仅一个模型，结果为特定模型观察；引导敏感性在所测模型间非单调，高度依赖模型类型（聊天或推理）。我们提出六标签故障分类法，显示高能力模型失败主因是格式违规，低能力模型则以文件错误为主，并提出面向层级的引导选择指南。
原文摘要 · Abstract (English)
A prevalent assumption in LLM agent deployment holds that more structured harnesses universally improve reliability, and that higher-capability models need proportionally less structural guidance -- together implying a monotone inverse relationship between model capability tier and optimal harness complexity. We test this hypothesis through a controlled 432-run experiment crossing six models across four capability tiers with three harness conditions (light, balanced, strict) on HEAT-24, a 24-task synthetic benchmark with git-based workspace verification. Our results refute the monotone inverse relationship on two fronts. First, for the frontier chat model evaluated (Gemini 2.5 Flash), increased harness verbosity lowers VTSR by 29-38 percentage points -- a harness-complexity paradox. Second, for the frontier reasoning model evaluated (Qwen3.5-122B, extended thinking enabled), strict harness achieves the highest VTSR (91.7%) and the lowest latency, the opposite of the prediction. Within the constrained tier, a 2B model (Gemma4:e2B) matches strong-open-tier stability at 91.7% across all harnesses. Because each tier is represented by a single model in this study, these results should be interpreted as model-specific observations; harness sensitivity appears non-monotone across the models evaluated, and depends critically on model type (chat vs. reasoning). We introduce a six-label failure taxonomy showing that format_violation dominates capable-model failures while wrong_file dominates low-capability failures, and we derive practical tier-aware harness selection guidelines.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。