arXiv:2505.24197cs.AI2025-05EMNLP被引 2

让智能体从实例演示中直接学用API,无需文档也能高效调用。

Learning API Functionality from In-Context Demonstrations for Tool-based Agents

  • 通过上下文演示学习API功能,不依赖传统文档。
  • 明确函数调用和自然语言评注可显著提升任务成功率。
  • 适合研究无文档环境下自进化智能体的开发者。

基于大语言模型的工具型智能体常依赖外部API文档来理解功能,但文档往往缺失、过时、私有或不一致,阻碍了可靠通用智能体的发展。本文提出一种新方向:直接从上下文演示中学习API功能,适用于无文档场景。我们使用3个数据集和6种模型,在专家代理与自探索生成的演示上进行实验,研究演示数量、LLM生成摘要与评估对任务成功率的影响。结果表明,即使最先进的大模型仍难以完全掌握此任务。提供显式函数调用和自然语言批评能显著提升参数填充准确性,从而提高成功率。我们分析了失败模式,识别出主要错误来源,并指出了未来在无文档、自我改进型API智能体方面的关键挑战。

原文摘要 · Abstract (English)

Digital tool-based agents, powered by Large Language Models (LLMs), that invoke external Application Programming Interfaces (APIs) often rely on documentation to understand API functionality. However, such documentation is frequently missing, outdated, privatized, or inconsistent-hindering the development of reliable, general-purpose agents. In this work, we propose a new research direction: learning of API functionality directly from in-context demonstrations. This task is a new paradigm applicable in scenarios without documentation. Using API benchmarks, we collect demonstrations from both expert agents and from self-exploration. To understand what information demonstrations must convey for successful task completion, we extensively study how the number of demonstrations and the use of LLM-generated summaries and evaluations affect the task success rate of the API-based agent. Our experiments across 3 datasets and 6 models show that learning functionality from in-context demonstrations remains a non-trivial challenge, even for state-of-the-art LLMs. We find that providing explicit function calls and natural language critiques significantly improves the agent's task success rate due to more accurate parameter filling. We analyze failure modes, identify sources of error, and highlight key open challenges for future work in documentation-free, self-improving, API-based agents.

API学习智能体LLM无文档

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。