arXiv:2607.23722cs.AI2026-07被引 1

评测大模型在真实产品场景中多步调用工具的能力,发现当前模型仍难可靠完成复杂任务。

E-Bench: Benchmarking Multi-Step Tool-Use Agents in Real-World Product Scenarios

论文配图:E-Bench: Benchmarking Multi-Step Tool-Use Agents in Real-World Product Scenarios
图 1 · 摘自论文原文
  • 构建可复用的合成环境与任务,分离环境生成与任务设计
  • 11个主流模型在323个任务中,最高通过率不足60%(Pass^3)
  • 适合评估真实应用中多步工具调用的智能体,尤其关注可靠性

大型语言模型正被用于代理,在多步骤交互中与有状态环境协作:获取隐藏信息、组合工具调用并修改状态。我们称此能力为多步工具使用。现有基准大多聚焦孤立API调用、短轨迹或难以扩展和控制的场景。我们提出E-Bench,一个完全合成的基准,涵盖三个产品领域(王者荣耀、QQ音乐、腾讯会议)共323个状态变更任务。E-Bench将环境合成与任务合成解耦:图引导的数据库填充构建可复用、无孤岛的产品环境;生成器-求解器异构性设计出同时存在信息缺口与工具缺口的任务,要求代理发现隐藏数据并组合多个工具调用才能改写状态。结果通过数据库状态差分确定性评分。由于环境与任务均为合成,E-Bench可在环境层面可控,在任务层面可扩展。对11个前沿大模型的基准测试显示,多步工具使用仍具挑战:最强模型的Pass^3低于60%;即使在支持代码执行的E-Bench-Code扩展中,可靠性(Pass^3)仍低于70%。

原文摘要 · Abstract (English)

Large Language Models (LLMs) are increasingly deployed as agents that interact with stateful environments over multiple steps: gathering hidden information, composing tool calls, and committing state changes. We refer to this capability as multi-step tool use. Existing benchmarks have advanced tool-use agent evaluation, but often focus on isolated API calls, short trajectories, or settings that are difficult to scale or control. We introduce E-Bench, a fully synthetic benchmark with 323 state-changing tasks across three product domains: Honor of Kings, QQ Music, and Tencent Meeting. E-Bench decouples environment synthesis from task synthesis: graph-guided database filling builds reusable, orphan-free product environments, while generator-solver asymmetry creates tasks with both an information gap and a tool gap, requiring agents to discover hidden data and compose multiple tool calls before changing state. Outcomes are graded deterministically by database-state diffs. Since both environments and tasks are synthetic, E-Bench is controllable at the environment level and scalable at the task level. Benchmarking 11 cutting-edge LLMs shows that multi-step tool use remains challenging: Pass^3 stays below 60% for the strongest models, and even with code execution in the E-Bench-Code extension, reliability (Pass^3) remains below 70%.

多步推理工具调用基准评测智能体

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。