构建稳定评估框架，测试大模型用API解决真实任务的能力
SEAL: Suite for Evaluating API-use of LLMs
- 用GPT-4模拟器+缓存解决真实API波动问题
- 覆盖调用、规划、响应全流程，支持多步骤推理
- 适合研究或部署大模型与外部系统交互的团队
大型语言模型在需要实时调用外部API的任务上存在局限。尽管已有ToolBench和APIGen等基准测试，但普遍存在泛化能力差、多步推理覆盖不足及因实时API波动导致结果不稳定的问题。本文提出SEAL，一个端到端的测试平台，用于评估大模型在真实世界中使用API的能力。SEAL统一现有基准，集成代理系统以测试API检索与规划，并通过GPT-4驱动的API模拟器配合缓存机制，实现确定性评估，克服实时接口波动。该测试平台涵盖API检索、调用及最终响应全过程，提供结构化性能对比的可靠框架。SEAL已开源，将持续更新新基准。
原文摘要 · Abstract (English)
Large language models (LLMs) have limitations in handling tasks that require real-time access to external APIs. While several benchmarks like ToolBench and APIGen have been developed to assess LLMs' API-use capabilities, they often suffer from issues such as lack of generalizability, limited multi-step reasoning coverage, and instability due to real-time API fluctuations. In this paper, we introduce SEAL, an end-to-end testbed designed to evaluate LLMs in real-world API usage. SEAL standardizes existing benchmarks, integrates an agent system for testing API retrieval and planning, and addresses the instability of real-time APIs by introducing a GPT-4-powered API simulator with caching for deterministic evaluations. Our testbed provides a comprehensive evaluation pipeline that covers API retrieval, API calls, and final responses, offering a reliable framework for structured performance comparison in diverse real-world scenarios. SEAL is publicly available, with ongoing updates for new benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。