对比三种代码执行方式，发现最省钱的工具组合取决于任务类型和模型设计。
When Does Restricting a Coding Agent to execute_code Help? A Regime $\times$ Agent-Design Ablation
- 固定模型和提示，只改变代码执行方式，进行四组对照实验。
- 在多数情况下，仅用代码执行工具比多工具更省成本，成功率相当。
- 成本优势主要来自缓存优化，而非提升通过率，适合关注效率的研究者。
现代编程代理提供多种工具接口——IDE 原语、bash 和 MCP 代码执行——但学界对哪种最重要存在三类矛盾观点。本文首次开展交叉对比：在合成计算任务与 SWE-bench Mini 修改任务上，保持模型、测试框架与提示不变，使用两个代理（Claude Code、OpenAI Codex CLI），对基础版、仅 bash、仅 execute_code 三种配置进行完整三臂消融实验，覆盖任务范式与代理设计双重维度。在四个（范式, 代理）组合中，限制为单一 execute_code 工具在三个组合中成本更低或无显著差异（Artifact/Claude 与 SWE-bench/Codex 显著更优；Artifact/Codex 方向性更优），且通过率在各组内统计等价。唯一例外是 SWE-bench/Claude，其仅代码模式成本高 14.4%（不显著），经局部成本分析发现源于失败轨迹的惩罚成本，而非成功运行时的每编辑开销。结论：最经济的工具表面由任务范式与代理设计共同决定，而非任一因素单独决定；成本信号应基于缓存调整后的成本，而非通过率——后者在当前模型规模下跨工具表面基本不变。基准测试框架、任务集与分析代码已开源：https://github.com/hyang0129/onlycodes。
原文摘要 · Abstract (English)
Modern coding agents expose multiple tool surfaces -- IDE primitives, bash, and Model Context Protocol (MCP) code-execution -- and the field has shipped three contradictory claims about which one matters. We run the missing crossed comparison: an integrity-clean three-arm ablation (baseline / bash_only / code_only) on synthetic computation tasks and SWE-bench Mini modification tasks, holding model, harness, and prompts fixed, with two agents (Claude Code, OpenAI Codex CLI) so the comparison spans both regime and agent-design axes. Across the four resulting (regime, agent) cells, restricting the agent to a single execute_code MCP tool is cheaper than -- or statistically tied with -- its cheapest tool-rich rival in three cells (significantly on Artifact/Claude and SWE-bench/Codex; directionally on Artifact/Codex), with pass rates statistically tied within each cell. The lone exception is SWE-bench/Claude, where code_only is directionally costlier (+14.4%, not significant); a conditional-cost analysis localizes that gap to failure-cost on doomed-run trajectories, not a per-edit tax on successful runs. Two implications: the cheapest tool surface is jointly determined by task regime and agent design rather than by either axis alone, and the headline cost signal lives in cache-adjusted cost -- not pass rate, which is invariant across surfaces at the model sizes we evaluate. The benchmark harness, task suite, and analysis code are available at https://github.com/hyang0129/onlycodes.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。