arXiv:2411.19804cs.SEcs.AI2024-11被引 8

用RAG和智能分块优化API集成,让大模型更高效地发现接口。

Advanced System Integration: Analyzing OpenAPI Chunking for Retrieval-Augmented Generation

  • 用RAG结合分块处理OpenAPI文档,减少输入token数。
  • 分块策略中基于LLM和格式的方法效果优于简单切分,召回率超90%。
  • 引入发现代理按需获取细节,提升检索精度与整体性能。

集成多个(子)系统是构建先进信息系统(ISs)的关键,尤其在动态环境下的全生命周期集成面临挑战。传统方式依赖注册表提供系统端点的API文档,而大型语言模型(LLMs)可据此自动生成系统集成(如服务组合),但受限于输入令牌长度,需对详尽的API描述进行精简。当前尚不清楚最优的预处理方法。本文(i)分析了检索增强生成(RAG)在端点发现中的应用及OpenAPI分块策略,以降低输入长度并保留关键信息;提出(ii)一种发现代理,仅接收最相关端点摘要,并按需检索详细信息,进一步压缩提示词长度并提升端点检索效率。在RestBench基准上评估不同分块方式与参数的端点检索召回率、精确率和F1分数,结果显示基于LLM和格式特定的分块优于朴素切分方法,且达到高召回率(>90%)、高精确率与高F1值。使用发现代理后,任务被拆分为细粒度子任务,显著提升总体性能,在令牌数、精确率和F1分数上均有改善。尽管如此,仍需进一步研究以确保所有必要端点被完整检索。

原文摘要 · Abstract (English)

Integrating multiple (sub-)systems is essential to create advanced Information Systems (ISs). Difficulties mainly arise when integrating dynamic environments across the IS lifecycle. A traditional approach is a registry that provides the API documentation of the systems' endpoints. Large Language Models (LLMs) have shown to be capable of automatically creating system integrations (e.g., as service composition) based on this documentation but require concise input due to input token limitations, especially regarding comprehensive API descriptions. Currently, it is unknown how best to preprocess these API descriptions. Within this work, we (i) analyze the usage of Retrieval Augmented Generation (RAG) for endpoint discovery and the chunking, i.e., preprocessing, of OpenAPIs to reduce the input token length while preserving the most relevant information. To further reduce the input token length for the composition prompt and improve endpoint retrieval, we propose (ii) a Discovery Agent that only receives a summary of the most relevant endpoints and retrieves details on demand. We evaluate RAG for endpoint discovery using the RestBench benchmark, first, for the different chunking possibilities and parameters measuring the endpoint retrieval recall, precision, and F1 score. Then, we assess the Discovery Agent using the same test set. With our prototype, we demonstrate how to successfully employ RAG for endpoint discovery to reduce the token count. While revealing high values for recall, precision, and F1, further research is necessary to retrieve all requisite endpoints. Our experiments show that for preprocessing, LLM-based and format-specific approaches outperform naïve chunking methods. Relying on an agent further enhances these results as the agent splits the tasks into multiple fine granular subtasks, improving the overall RAG performance in the token count, precision, and F1 score.

RAGAPI集成LLM服务组合

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。