评测大模型在长上下文下的多步函数调用能力
ComplexFuncBench: Exploring Multi-Step and Constrained Function Calling under Long-Context Scenario
- 构建多步骤约束函数调用的评测基准，支持128k上下文
- 发现顶尖大模型在复杂任务中仍存在显著调用缺陷
- 适合研究模型推理与真实世界API集成的学者
为提升大语言模型（LLMs）调用实时API生成更准确、更新响应的能力，本文提出ComplexFuncBench，一个涵盖五个真实场景的复杂函数调用评测基准。相较于现有工作，该基准包含多步与约束性函数调用，要求长参数填写、参数值推理及128k长上下文处理能力。同时，我们设计了自动评估框架ComplexEval，实现对复杂函数调用任务的量化评估。通过全面实验，揭示了当前最优大模型在函数调用上的不足，并指明未来优化方向。数据与代码已开源。
原文摘要 · Abstract (English)
Enhancing large language models (LLMs) with real-time APIs can help generate more accurate and up-to-date responses. However, evaluating the function calling abilities of LLMs in real-world scenarios remains under-explored due to the complexity of data collection and evaluation. In this work, we introduce ComplexFuncBench, a benchmark for complex function calling across five real-world scenarios. Compared to existing benchmarks, ComplexFuncBench encompasses multi-step and constrained function calling, which requires long-parameter filing, parameter value reasoning, and 128k long context. Additionally, we propose an automatic framework, ComplexEval, for quantitatively evaluating complex function calling tasks. Through comprehensive experiments, we demonstrate the deficiencies of state-of-the-art LLMs in function calling and suggest future directions for optimizing these capabilities. The data and code are available at \url{https://github.com/THUDM/ComplexFuncBench}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。