为智能体设计资源约束下的多维评估框架，揭示高准确率未必适合实际部署。
AgentSLABench: Evaluating and Benchmarking Agentic Systems Under Resource Constraints
- 构建包含16个任务的资源感知评测平台，量化智能体的正确性与资源消耗。
- 专用智能体在5个核心任务中3项成功率100%，通用模型在4个任务上完全失败。
- 提出效率调整成功率（EASR），强调资源消耗与性能需综合考量。
我们提出AgentSLABench，一个面向自主AI智能体的资源感知评估框架，可在声明的资源预算下，同时测量正确性、延迟、成本、计算、内存和网络使用。不同于仅报告准确率的标准基准，AgentSLABench为每个智能体在每项任务中生成多维性能画像，类比系统剖析工具（如perf、pprof）对代码资源消耗的度量，但增加任务正确性作为首要维度。该框架涵盖6大类共16个任务环境（5个核心：多跳问答、零售替代、代码生成、网页购物、旅行规划；11个扩展），采用隔离Docker容器、声明式CPU/内存/时间/网络预算、带SHA256哈希的密封测试集及标准化剖析协议。我们对5个通用基线智能体（ReAct、PlanAndSolve、Reflexion、CoT、Random）和4个任务专用智能体进行评测，发现专用智能体在3/5个核心任务（fact_qa、web_shopping、travel_planning）中实现100%成功，零售和code_gen任务成功率66.7%-83.3%，而通用基线在4/5领域任务上完全失败。关键的是，我们引入效率调整成功率（EASR）——以相对于声明预算的资源消耗加权的成功率，揭示高准确率但无资源限制的性能不可用于生产。我们公开完整基础设施、密封测试集及剖析结果，支持可复现的资源感知智能体评估。
原文摘要 · Abstract (English)
We present AgentSLABench, a resource-aware evaluation framework for autonomous AI agents that measures correctness alongside latency, cost, compute, memory, and network usage under declared resource budgets. Unlike standard benchmarks that report only accuracy, AgentSLABench produces a multi-dimensional profile per agent per task - the same way systems profilers (perf, pprof, cProfile) measure resource consumption of code, but extended with task correctness as a first-class dimension. AgentSLABench provides 16 task environments across 6 categories (5 core: multi-hop QA, retail substitution, code generation, web shopping, travel planning; 11 extended) with isolated Docker containers, declared CPU/memory/time/network budgets, sealed test sets with SHA256 hashes, and a standardized profiling protocol. We profile 5 general-purpose baseline agents (ReAct, PlanAndSolve, Reflexion, CoT, Random) plus 4 task-specialized agents, finding that specialized agents achieve 100% success on 3/5 core tasks (fact_qa, web_shopping, travel_planning) and 66.7-83.3% on retail and code_gen, while general baselines fail entirely on 4/5 domain tasks. Crucially, we report the Efficiency-Adjusted Success Rate (EASR) - success weighted by resource consumption relative to declared budgets - revealing that high accuracy at unbounded cost is not production-viable. We release the full infrastructure, sealed test sets, and profiling results to enable reproducible, resource-aware agent evaluation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。