评测大模型生成网页API调用代码的能力，发现普遍错误率高。
Benchmarking Web API Integration Code Generation
- 构建WAPIIBench数据集与评估流程，专门测试API调用代码生成。
- 开源大模型在任务中最高仅40%正确率，常见错误包括接口幻觉。
- 适合关注AI编程工具局限性的开发者和研究者参考。
API集成是数字基础设施的核心，使软件系统得以连接与交互。然而，诸多研究表明，编写或生成正确的API调用代码，尤其是网络API，极具挑战性。尽管大型语言模型（LLMs）在软件开发中日益流行，其在自动化生成网络API调用代码方面的效果仍缺乏研究。为此，本文提出WAPIIBench——一个用于评估LLM生成网络API调用代码能力的数据集与评估流程。对多个开源LLM的实验表明，生成API调用面临巨大挑战，普遍存在接口幻觉、参数使用错误等问题。在所有评测任务中，无一开源模型解决超过40%的任务。
原文摘要 · Abstract (English)
API integration is a cornerstone of our digital infrastructure, enabling software systems to connect and interact. However, as shown by many studies, writing or generating correct code to invoke APIs, particularly web APIs, is challenging. Although large language models (LLMs) have become popular in software development, their effectiveness in automating the generation of web API integration code remains unexplored. In order to address this, we present WAPIIBench, a dataset and evaluation pipeline designed to assess the ability of LLMs to generate web API invocation code. Our experiments with several open-source LLMs reveal that generating API invocations poses a significant challenge, resulting in hallucinated endpoints, incorrect argument usage, and other errors. None of the evaluated open-source models was able to solve more than 40% of the tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。