测试大模型在有限预算下智能分配工具的能力，发现主流模型在编程任务中表现不佳。
AllocBench: Measuring Online Tool Allocation Capability in LLM Agents
- 设计双场景基准，测试模型在抽象与代码任务中的工具分配策略
- 前沿模型在抽象任务接近最优，但在代码生成中普遍过度创建工具
- 仅GPT-5.6 Sol保持选择性，开源模型可在抽象任务泛化但无法迁移至代码
构建一个配对基准，测试大模型代理在固定预算下是否具备有意识的工具分配行为，涵盖抽象文本任务与代码构建任务。实验发现，所有测试的前沿模型（Claude Haiku、Claude Opus、GPT-5.4-mini、GPT-5.6 Sol）在抽象任务中表现接近最优，但在代码任务中无法迁移该能力。进一步分析显示，前三个模型即使在不评估脚本时也出现过度创建工具的问题，而GPT-5.6 Sol仅在完整构造阶段才失效。此外，一个针对抽象任务训练的开源Qwen模型可泛化至未见词汇变体，但在脚本分配任务中无提升。结果表明，线上工具分配是现代前沿模型的关键能力边界。
原文摘要 · Abstract (English)
Creating a reusable tool is an investment: an agent pays a fixed cost now in exchange for the potential of future reuse. Therefore, a user should prefer an agent that creates a small number of highly reusable tools, rather than many one-offs. We introduce a paired benchmark that tests whether LLM agents exhibit conscious allocation behavior under a fixed budget in two contexts: an abstract text-based formulation and a code-construction task. We find that every frontier model we test---Claude Haiku, Claude Opus, GPT-5.4-mini, and GPT-5.6 Sol---acts near-optimally in the abstract framing but fails to transfer this ability to script-writing. Through further experiments, we identify the particular failure modes for each model. Notably, the first three models fail even when the scripts are not evaluated, while GPT-5.6 Sol stays selective under that weaker manipulation and collapses only at full construction. Furthermore, an open-source Qwen model policy-trained for abstract allocation generalizes this ability across held-out lexical variations, but sees no improvement at script allocation. Together, these results establish online tool allocation as a significant capability boundary, even for modern frontier models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。