测试大模型代理在预算约束下的经济决策能力，发现完成任务不等于省钱。
EcoAgent-Bench: Evaluating Economic Decision-Making in Budget-Constrained LLM Agents
- 设计带价格和预算的任务基准，强制代理权衡资源使用与任务目标。
- 多数代理要么过早放弃、要么浪费预算，经济一致性最高仅7.3%。
- 适合研究智能体资源管理、自动化决策或系统优化的学者与工程师。
传统代理评估只关注任务完成率，忽略资源消耗。实际部署中，选择本地查询、广泛搜索、组合工具、更强模型或人工升级本身就是任务的一部分。我们提出EcoAgent-Bench，每个任务均设定动作价格与明确预算。该基准包含304个源自真实场景的任务，覆盖五个类别，改编自GAIA、HotpotQA和MuSiQue，测试四类关键决策：避免无谓升级、本地证据不足时升级、选择模型层级、在无法支持前提下停止。我们在工具-API与工作区-CLI两种设置下评估七种LLM代理及四种脚本化基准。微平均准确率显示单边策略占优：始终升级的基准表现高，但在节约型任务中失败。因此引入经济一致性分数（升级导向与节约导向任务组准确率中的较低值），揭示此缺陷。工具-API代理微严格成功率仅为3.9%-24.0%，经济一致性最高7.3%，常在未必要时停止或在廉价任务上过度支出。阈值扫描显示，GPT-5.4的升级率从0%增至仅3%。结果表明，预算内完成任务与经济合理行动是两个独立属性。我们公开任务包、转换管道、冻结评估环境及完整结果文件，以支持深入研究。
原文摘要 · Abstract (English)
Agent benchmarks usually measure task completion and treat resource use as an auxiliary statistic. In deployment, however, the choice among a local lookup, broad search, composite research tool, stronger model, or human escalation is part of the task itself. We introduce EcoAgent-Bench, in which every task specifies priced actions and an explicit budget. Its 304 real-derived tasks span five families adapted from GAIA, HotpotQA, and MuSiQue, and test four decisions: avoiding unnecessary escalation, escalating when local evidence is insufficient, selecting a model tier, and stopping on unsupported premises. We evaluate seven LLM agents in tool-API and workspace-CLI settings, together with four oracle scripted controls. Micro-averaged accuracy rewards one-sided policies: always-escalate controls achieve high micro success while failing save-oriented tasks. We therefore also report an economic-consistency score (the worse of accuracy on upgrade-oriented and save-oriented family groups) which exposes this failure. Tool-API agents attain only 3.9-24.0% micro strict success (at most 7.3% economic consistency), often either stopping before warranted escalation or overspending on cheap tasks. A threshold-crossing budget sweep changes GPT-5.4's escalation rate from 0% to only 3%. These results show that completion under a budget and economical action selection are distinct properties. We release the task bundle, transformation pipeline, frozen evaluation environments, and integrity-bound result artifacts needed to study both.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。