构建真实工具对话数据集,提升大模型多轮协作能力。
ToolDial: Multi-turn Dialogue Generation Method for Tool-Augmented Language Models
- 基于RapidAPI接口构建1.1万条多轮对话,每条平均8.95轮
- 引入16种用户与系统交互动作,模拟真实信息缺失场景
- 提出API兼容图生成方法,助力模型自动调用必要接口
工具增强型语言模型(TALM)通过外部API在多个领域响应用户查询。然而,现有基准数据集中的对话过于简单,无法反映真实场景,如模型需主动澄清问题或在信息缺失时调用额外API。为此,我们构建并发布了ToolDial数据集,包含11,111条多轮对话,平均每条8.95轮,基于RapidAPI的API。该数据集具有两大特点:一是对话中包含16种用户与系统行为(如“请求”、“澄清”、“失败提示”),以捕捉真实交互动态;二是模拟系统根据API文档向用户请求必要信息,并在用户未提供时主动寻找其他可用API。为支持此过程,我们提出一种生成API图的方法,表示不同API之间的输入输出兼容性。使用ToolDial,我们评估了多种语言模型在预测正确动作和从对话历史中提取API参数值方面的表现,结果显示现代模型准确率低于70%,仍有巨大提升空间。数据集与代码已开源至https://github.com/holi-lab/ToolDial。
原文摘要 · Abstract (English)
Tool-Augmented Language Models (TALMs) leverage external APIs to answer user queries across various domains. However, existing benchmark datasets for TALM research often feature simplistic dialogues that do not reflect real-world scenarios, such as the need for models to ask clarifying questions or proactively call additional APIs when essential information is missing. To address these limitations, we construct and release ToolDial, a dataset comprising 11,111 multi-turn dialogues, with an average of 8.95 turns per dialogue, based on APIs from RapidAPI. ToolDial has two key characteristics. First, the dialogues incorporate 16 user and system actions (e.g., "Request", "Clarify", "Fail inform") to capture the rich dynamics of real-world interactions. Second, we simulate dialogues where the system requests necessary information from the user based on API documentation and seeks additional APIs if the user fails to provide the required information. To facilitate this process, we introduce a method for generating an API graph that represents input and output compatibility between APIs. Using ToolDial, we evaluate a suite of language models on their ability to predict correct actions and extract input parameter values for API calls from the dialogue history. Modern language models achieve accuracy scores below 70%, indicating substantial room for improvement. We release our dataset and code at https://github.com/holi-lab/ToolDial.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。