压缩代码生成的令牌数不等于节省成本,反而可能增加开销。
Token Reduction Is Not Cost Reduction
- 通过控制实验对比三种压缩方法与原始模型
- 最大压缩减少输出令牌38.4%,但账单成本反升6.8%
- 优化应关注任务成功率和整体成本,而非仅令牌数
针对代码代理的令牌压缩工具常以减少令牌数量为评估标准,但令牌数本身并不决定端到端推理成本。我们在可控编码任务中,将三种令牌压缩方法与未修改的Claude Code基线进行对比,测量了服务商计费成本、任务成功率、缓存流量及代理行为。最大压缩设置使交付的工具输出令牌减少38.4%,但计费成本反而上升6.8%;较轻压缩仅带来微小且统计上不确定的节省。跨任务分析显示,令牌减少与成本降低相关性很弱(皮尔逊相关系数r=0.15)。成本分解表明,提示缓存的创建与读取主导输入侧成本,仅有有限部分支出可通过工具输出压缩直接优化。此外,压缩可能通过额外检索、诊断、测试和交互轮次改变代理路径,抵消局部令牌节省。在SWE-bench Go子集上,激进压缩还降低了成功补丁应用率。结果表明,在工具密集型代码代理中,令牌减少不能作为成本减少的可靠代理。因此,有效优化应以每成功任务的成本为评价基准,综合考虑缓存行为、轨迹变化与正确性,而非仅依赖令牌数量。
原文摘要 · Abstract (English)
Token-reduction tools for coding agents are often evaluated by the number of tokens they remove, but token count alone does not determine end-to-end inference cost. We evaluate three token-reduction approaches against an unmodified Claude Code baseline across controlled coding tasks, measuring provider-billed cost, task success, cache traffic, and agent behavior. The largest compression setup reduced delivered tool-output tokens by 38.4% but increased billed cost by 6.8%, while lighter compression produced only small and statistically uncertain savings. Across tasks, token reduction was weakly correlated with cost reduction (Pearson r = 0.15). Cost decomposition shows that prompt-cache creation and reads dominate the measured input-side cost, leaving only a limited fraction of total spend directly addressable by tool-output compression. We also find that compression can alter agent trajectories through additional retrieval, diagnosis, testing, and turns, offsetting local token savings. On a SWE-bench Go subset, aggressive compression also reduced successful patch application. These results show that token reduction is not a reliable proxy for cost reduction in tool-heavy coding agents. Effective optimization should therefore be evaluated at the level of cost per successful task, including cache behavior, trajectory changes, and correctness rather than token counts alone.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。