为大模型代理测试企业API工具提供系统化方案
A Framework for Testing and Adapting REST APIs as LLM Tools
- 生成数据驱动的测试用例，自动转换为自然语言指令
- 发现2400+测试案例中的输入误解、输出失败等错误
- 支持调试与工具优化，适合企业级API自动化场景
大型语言模型（LLMs）正被用于构建能执行复杂任务的自主代理，这些任务常依赖企业系统中的外部API。直接使用这些API存在输入模式复杂、响应冗长等问题。现有基准未能涵盖这些挑战，导致评估企业API在代理驱动自动化中的可用性存在空白。本文提出一个测试框架，系统评估企业API作为LLM代理的Python工具时的表现。该框架生成数据感知的测试用例，将其转化为自然语言指令，并评估代理是否能正确调用工具、处理输入并解析输出。我们在多个领域生成超过2400个测试用例，建立常见错误分类体系，包括输入误解、输出失败和模式不匹配。进一步对错误进行分类，以支持调试与工具改进。本框架为使企业API成为代理应用的可靠工具提供了系统方法。
原文摘要 · Abstract (English)
Large Language Models (LLMs) are increasingly used to build autonomous agents that perform complex tasks with external tools, often exposed through APIs in enterprise systems. Direct use of these APIs is difficult due to the complex input schema and verbose responses. Current benchmarks overlook these challenges, leaving a gap in assessing API readiness for agent-driven automation. We present a testing framework that systematically evaluates enterprise APIs when wrapped as Python tools for LLM-based agents. The framework generates data-aware test cases, translates them into natural language instructions, and evaluates whether agents can correctly invoke the tool, handle their inputs, and process its responses. We apply the framework to generate over 2400 test cases across different domains and develop a taxonomy of common errors, including input misinterpretation, output failures, and schema mismatches. We further classify errors to support debugging and tool refinement. Our framework provides a systematic approach to enabling enterprise APIs as reliable tools for agent-based applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。