构建多跳工具调用评测数据集，揭示大模型在复杂任务中的能力瓶颈。
ToolHop: A Query-Driven Benchmark for Evaluating Large Language Models in Multi-Hop Tool Use
- 基于查询驱动构建包含3912个工具的评测数据集
- 顶尖模型GPT-4o准确率仅49.04%，暴露严重短板
- 提供可执行工具与反馈机制，适合评估工具链推理能力
高效评估多跳工具使用对分析大语言模型的理解、推理与函数调用能力至关重要。然而，可靠评测数据集的缺失阻碍了进展。为此，我们提出ToolHop，一个包含995个用户查询和3912个相关工具的数据集，专为严格评估多跳工具使用而设计。通过新颖的查询驱动数据构建方法（包括工具创建、文档优化与代码生成），ToolHop确保查询多样性、工具间有意义依赖、本地可执行性、详细反馈及可验证答案。我们在五个模型家族（LLaMA3.1、Qwen2.5、Gemini1.5、Claude3.5、GPT）上评估了14个LLM，发现处理多跳工具使用场景存在显著挑战。领先模型GPT-4o准确率为49.04%，表明仍有巨大提升空间。进一步分析揭示不同模型家族在工具使用策略上的差异，为开发更有效方法提供关键洞察。代码与数据见https://huggingface.co/datasets/bytedance-research/ToolHop。
原文摘要 · Abstract (English)
Effective evaluation of multi-hop tool use is critical for analyzing the understanding, reasoning, and function-calling capabilities of large language models (LLMs). However, progress has been hindered by a lack of reliable evaluation datasets. To address this, we present ToolHop, a dataset comprising 995 user queries and 3,912 associated tools, specifically designed for rigorous evaluation of multi-hop tool use. ToolHop ensures diverse queries, meaningful interdependencies, locally executable tools, detailed feedback, and verifiable answers through a novel query-driven data construction approach that includes tool creation, document refinement, and code generation. We evaluate 14 LLMs across five model families (i.e., LLaMA3.1, Qwen2.5, Gemini1.5, Claude3.5, and GPT), uncovering significant challenges in handling multi-hop tool-use scenarios. The leading model, GPT-4o, achieves an accuracy of 49.04%, underscoring substantial room for improvement. Further analysis reveals variations in tool-use strategies for various families, offering actionable insights to guide the development of more effective approaches. Code and data can be found in https://huggingface.co/datasets/bytedance-research/ToolHop.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。