评测大模型调用API能力，提出新数据集与混合路由方法
CallNavi, A Challenge and Empirical Study on LLM Function Calling and Routing
- 构建专用数据集，评估模型选接口、填参数和嵌套调用能力
- 发现复杂任务下主流模型接口生成准确率下降超40%
- 结合通用大模型与微调+提示工程，提升参数生成质量
基于API的聊天机器人系统在软件工程中日益重要，但其有效性依赖于准确生成和执行API调用。尤其在需要多步交互、复杂参数化及嵌套依赖的场景下挑战显著。本文通过三大贡献推进AI驱动的软件开发评估：(1) 提出一个专为基准测试API函数选择、参数生成和嵌套执行而设计的新数据集；(2) 对当前最先进的语言模型进行实证评估，分析其在不同任务复杂度下的接口生成与参数准确性表现；(3) 提出一种混合式API路由方案，结合通用大模型进行接口选择，辅以微调模型与提示工程实现参数生成。该方法显著提升了聊天机器人系统的API执行效果，为实际软件工程中的设计、测试与运维流程提供了可落地的方法论。
原文摘要 · Abstract (English)
API-driven chatbot systems are increasingly integral to software engineering applications, yet their effectiveness hinges on accurately generating and executing API calls. This is particularly challenging in scenarios requiring multi-step interactions with complex parameterization and nested API dependencies. Addressing these challenges, this work contributes to the evaluation and assessment of AI-based software development through three key advancements: (1) the introduction of a novel dataset specifically designed for benchmarking API function selection, parameter generation, and nested API execution; (2) an empirical evaluation of state-of-the-art language models, analyzing their performance across varying task complexities in API function generation and parameter accuracy; and (3) a hybrid approach to API routing, combining general-purpose large language models for API selection with fine-tuned models and prompt engineering for parameter generation. These innovations significantly improve API execution in chatbot systems, offering practical methodologies for enhancing software design, testing, and operational workflows in real-world software engineering contexts.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。