不同编程语言下代码代理的耗token差异巨大,揭示了语言对效率的关键影响。
The Best Programming Language for Tokenmaxxing: An Investigation of Coding Agent Behavior Across Programming Languages
- 通过控制题目难度,对比五种模型在四种语言中的token消耗
- 同一问题在不同语言中最高差达3.5倍,且结果一致跨模型
- 发现代理在陌生语言中反复生成无法编译的代码,偏好用Python原型
尽管代码代理在多种编程语言中表现高效,本文首次揭示语言对耗token量有显著影响。我们在Python、Java、Rust和OCaml上评估了五种近期模型,严格控制问题难度,发现各语言间存在显著且跨模型一致的token消耗差异。为理解原因,我们重新执行每个中间解,将轨迹抽象为测试结果向量序列,并标注解决方案间的操作。结果显示,代理在不熟悉语言中频繁生成无法编译的代码,且会反复修改已通过的解。进一步分析轨迹文本发现,代理常在注释中规划代码,不信任给定测试而自行构造输入,并倾向于用Python进行原型开发以规避陌生语言。结果表明,按语言的token效率应成为多语言代理评估与开发的重要指标,也为准则制定者提供成本参考。
原文摘要 · Abstract (English)
Although coding agents are now very effective in a variety of programming languages, this paper first shows that the cost (in tokens) can very significantly by programming language. We evaluate five recent models on programming problems in Python, Java, Rust, and OCaml. We carefully control for problem difficulty, and show that there can be stark variation in token consumption that is consistent across models. To understand why, we analyze both the structure and content of agent trajectories. First, we re-execute every intermediate solution and abstract each trajectory as a sequence of test-outcome vectors, then label the work between successive solutions. This reveals agents repeatedly producing noncompiling solutions in unfamiliar languages and revising solutions that already pass. Second, we analyze trajectory text, finding that agents plan solutions in code comments, distrust the provided tests in favor of inputs they invent, and sidestep unfamiliar target languages by prototyping in Python. Our results show that by-language token efficiency is a metric that should be considered when benchmarking and developing multilingual agents, and, for the tokenmaxxer, a guide to the most expensive language to work in.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。