构建多轮多方对话工具调用评测基准，揭示现有模型实战能力不足。
DICE-BENCH: Evaluating the Tool-Use Capabilities of Large Language Models in Multi-Round, Multi-Party Dialogues
- 基于工具图与多智能体生成真实对话场景，保持多轮依赖关系。
- 19个大模型在新基准上表现普遍不佳，最高仅达32%正确率。
- 适合评估模型在复杂协作任务中的工具使用能力。
现有函数调用评测集中于单轮交互，忽略了现实场景的复杂性。为此，我们提出DICE-SCORE指标，量化对话中工具信息（如函数名、参数值）的分散程度。通过该指标分析现有基准，发现得分显著偏低，凸显现实场景缺失问题。为此，我们构建DICE-BENCH框架，利用工具图维持多轮依赖，并结合具有不同角色的多智能体系统提升对话自然度，合成高质量函数调用数据集。最终数据集包含1,607个高DICE-SCORE实例。在19个大语言模型上的实验表明，当前模型距离实际应用仍有显著差距。代码与数据已公开：https://snuhcc.github.io/DICE-Bench/。
原文摘要 · Abstract (English)
Existing function-calling benchmarks focus on single-turn interactions. However, they overlook the complexity of real-world scenarios. To quantify how existing benchmarks address practical applications, we introduce DICE-SCORE, a metric that evaluates the dispersion of tool-related information such as function name and parameter values throughout the dialogue. Analyzing existing benchmarks through DICE-SCORE reveals notably low scores, highlighting the need for more realistic scenarios. To address this gap, we present DICE-BENCH, a framework that constructs practical function-calling datasets by synthesizing conversations through a tool graph that maintains dependencies across rounds and a multi-agent system with distinct personas to enhance dialogue naturalness. The final dataset comprises 1,607 high-DICE-SCORE instances. Our experiments on 19 LLMs with DICE-BENCH show that significant advances are still required before such models can be deployed effectively in real-world settings. Our code and data are all publicly available: https://snuhcc.github.io/DICE-Bench/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。