构建2500+真实API环境，评测大模型多步调用能力
Live API-Bench: 2500+ Live APIs for Testing Multi-Step Tool Calling
- 将NL2SQL数据转为可交互API环境，覆盖三类任务形式
- 10个LLM任务完成率仅7%~47%，交互设置下提升至50%
- 适合评估大模型在真实企业场景下的工具调用能力
大型语言模型（LLMs）越来越多依赖外部工具和API来执行自然语言描述的复杂任务。在真实企业环境中评估此类工具调用能力面临挑战：API常为专有、异构且难以共享，限制了可复现的基准测试。为此，我们提出Live API Bench，一个通过将NL2SQL数据集转换为交互式API环境构建的综合性基准。其流水线将BIRD SQL中的SQL查询转化为三种形式（SLOT、SEL、REST）的可执行API序列，分别覆盖基础通用操作、领域特定多步任务和面向函数的RESTful交互。该基准涵盖11个数据库，包含超过2500个可调用工具，配套人工编写查询、真实API序列及验证后的最终答案。Live API Bench支持对工具使用核心挑战的系统性评估，包括错误处理、顺序推理、参数生成、响应解析及跨领域的鲁棒性。我们评估了10个LLM和4个ReACT代理，观察到任务完成率仅为7%至47%，在交互代理设置下小幅提升至50%，凸显出大模型工具调用性能仍有巨大提升空间。本文已发布所有代码与数据。
原文摘要 · Abstract (English)
Large language models (LLMs) increasingly rely on external tools and APIs to execute complex tasks specified in natural language. Evaluating such tool calling capabilities in realistic enterprise settings is challenging: APIs are often proprietary, heterogeneous, and difficult to share, limiting reproducible benchmarks. To address this, we introduce Live API Bench, a comprehensive benchmark constructed by transforming NL2SQL datasets into interactive API environments. Our pipeline converts SQL queries from BIRD SQL into executable API sequences across three formulations SLOT, SEL, and REST covering minimal general purpose operations, domain specific multi step tasks, and function oriented RESTful interactions, respectively. The benchmark spans 11 databases with over 2,500 invocable tools, paired with human authored queries, ground truth API sequences, and verified final answers. Live API Bench enables systematic evaluation of core challenges in tool use, including error handling, sequential reasoning, parameter generation, response parsing, and robustness across diverse domains. We evaluate 10 LLMs and 4 ReACT agents, observing low task completion rates (7 to 47pct), which improve modestly to 50pct under interactive agent settings, highlighting substantial scope for improving LLM tool calling performance. We release all code and data associated with this paper.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。