构建首个专家标注的API图数据集，提升工具代理的调用准确率。
In-N-Out: A Parameter-Level API Graph Dataset for Tool Agents
- 将API文档转为结构化依赖图，显式表达接口调用关系。
- 使用该数据集后，多工具查询生成性能接近翻倍提升。
- 适合研究大模型工具调用、API理解与自动化系统开发的人群。
工具代理——基于大语言模型与外部API交互的系统——能执行真实任务，但面对复杂任务时难以正确识别和有序调用API。为此，本文将API文档转化为结构化API图，捕捉接口依赖关系，并用于支持需要组合调用的多工具查询。我们提出In-N-Out，首个基于两个真实世界API基准及其文档构建的专家标注API图数据集。实验表明，使用In-N-Out显著提升工具检索与多工具查询生成性能，几乎使仅依赖文档的LLM性能翻倍。此外，基于In-N-Out微调的模型可缩小90%的性能差距，证明该数据集有助于模型学习理解API文档与参数关系。结果表明，显式API图对工具代理具有重要价值，In-N-Out是关键资源。数据与代码已开源：https://github.com/holi-lab/In-N-Out-API-Graph。
原文摘要 · Abstract (English)
Tool agents--LLM-based systems that interact with external APIs--offer a way to execute real-world tasks. However, as tasks become increasingly complex, these agents struggle to identify and call the correct APIs in the proper order. To tackle this problem, we investigate converting API documentation into a structured API graph that captures API dependencies and leveraging it for multi-tool queries that require compositional API calls. To support this, we introduce In-N-Out, the first expert-annotated dataset of API graphs built from two real-world API benchmarks and their documentation. Using In-N-Out significantly improves performance on both tool retrieval and multi-tool query generation, nearly doubling that of LLMs using documentation alone. Moreover, graphs generated by models fine-tuned on In-N-Out close 90% of this gap, showing that our dataset helps models learn to comprehend API documentation and parameter relationships. Our findings highlight the promise of using explicit API graphs for tool agents and the utility of In-N-Out as a valuable resource. We release our dataset and code at https://github.com/holi-lab/In-N-Out-API-Graph.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。