编码智能体效率受提示、推理努力与框架设计共同影响，不能只看令牌数。
Prompt-Induced Waste in Coding Agents: Reasoning, Effort, Harness Design, and End-to-End Cost
- 提出用任务成功率与总成本衡量编码代理效率，而非仅依赖令牌数量。
- 实验发现相同任务下不同提示会改变推理行为，额外计算未必提升效果。
- 强调需控制提示语义、推理强度和测试框架等变量进行公平评估。
编码智能体的效率无法仅通过令牌数量或模型价格来衡量。端到端成本与任务成功率取决于提示语义、推理努力、框架策略、模型选择、任务难度、工具使用、上下文管理及提供商计费方式。受控实验表明，同一任务下不同提示词可改变推理与验证行为；在复杂任务中增加推理努力可能带来帮助，但也可能徒增开销而无收益；效率优化的效果会随框架变化而改变。这些结果说明，提示、推理努力与框架是相互作用的实验因素，而非独立变量。我们以智能体轨迹引发的每成功任务成本来建模效率，令牌与缓存数量仅为轨迹的度量，不足以作为优化目标。因此，智能体评估应同时测量成功率与端到端成本，并控制决定轨迹生成的系统变量。
原文摘要 · Abstract (English)
Coding-agent efficiency cannot be characterized by token count or model price alone. End-to-end cost and task success depend jointly on prompt semantics, inference effort, harness policy, model, task difficulty, tool use, context management, and provider accounting. Controlled experiments show that prompt wording can change reasoning and verification behavior without changing the task, that additional inference effort can help on difficult tasks but can also add cost without benefit, and that the value of an efficiency intervention can change when the harness changes. These results show that prompt, effort, and harness are interacting experimental factors rather than independent controls. We model efficiency as cost per successful task induced by the agent trajectory. Token and cache counts are measurements of that trajectory, not sufficient optimization targets. Agent evaluations should therefore measure success and end-to-end cost while controlling the system variables that determine how the trajectory is produced.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。