评测大模型在嵌套API调用中的表现，发现最佳模型准确率仅28%。
NESTFUL: A Benchmark for Evaluating LLMs on Nested Sequences of API Calls
- 构建嵌套API调用基准测试NESTFUL，包含1800+可执行调用序列。
- GPT-4o在完整序列匹配上准确率为28%，胜率60%。
- 适合研究大模型工具调用与自主代理的学者使用。
基于大语言模型（LLMs）的自主代理在解决复杂现实任务中日益兴起，对模型调用外部工具、API和自定义函数的能力提出更高要求。当前研究虽关注工具调用，但评估基准和数据集未能跟上任务复杂性的发展。本文聚焦嵌套序列这一关键复杂性，提出NESTFUL基准，用于评估模型在嵌套API调用（即前一次调用输出作为下一次输入）上的表现。该基准包含1800多个可执行的嵌套序列。在多种模型上的实验表明，表现最佳的GPT-4o仅达到28%的完整序列匹配准确率和60%的胜率，显示该领域仍有巨大提升空间。结果分析为未来研究提供了方向，并已将数据集以Apache 2.0许可发布于https://github.com/IBM/NESTFUL。
原文摘要 · Abstract (English)
The resurgence of autonomous agents built using large language models (LLMs) to solve complex real-world tasks has brought increased focus on LLMs' fundamental ability of tool or function calling. At the core of these agents, an LLM must plan, execute, and respond using external tools, APIs, and custom functions. Research on tool calling has gathered momentum, but evaluation benchmarks and datasets representing the complexity of the tasks have lagged behind. In this work, we focus on one such complexity, nested sequencing, with the goal of extending existing benchmarks and evaluation. Specifically, we present NESTFUL, a benchmark to evaluate LLMs on nested sequences of API calls, i.e., sequences where the output of one API call is passed as input to a subsequent call. NESTFUL contains 1800+ nested sequences where all the function calls are executable. Experimental results on a variety of models show that the best-performing model (GPT-4o) achieves a full sequence match accuracy of 28% and a win-rate of 60%, necessitating a large scope for improvement in the nested sequencing aspect of function calling. Our analysis of these results provides possible future research directions for the community, in addition to a benchmark to track progress. We have released the NESTFUL dataset under the Apache 2.0 license at https://github.com/IBM/NESTFUL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。