用反思+检索增强修复kubectl命令执行失败问题
Repairing Tool Calls Using Post-tool Execution Reflection and RAG
- 通过大模型反思与工具文档RAG结合，自动分析并修正命令错误
- 使命令成功率提升55%，用户问题解决率平均提高36%
- 适合运维、开发人员使用，尤其在Kubernetes管理场景中
智能体系统通过调用工具（如Python函数、REST API或kubectl命令）与外部系统交互，但常因语法或语义错误导致失败。本文提出一种后执行反思机制，结合大语言模型反思与领域特定的检索增强生成（RAG），利用具体工具文档及故障排查资料进行错误修复。聚焦于Kubernetes中的kubectl工具，实证研究显示，该方法使55%评估模型的命令执行成功率提升，平均问题解决率提高36%；相比官方文档，使用故障排查文档可使成功率再提升10%。
原文摘要 · Abstract (English)
Agentic systems interact with external systems by calling tools such as Python functions, REST API endpoints, or command line tools such as kubectl in Kubernetes. These tool calls often fail for various syntactic and semantic reasons. Some less obvious semantic errors can only be identified and resolved after analyzing the tool's response. To repair these errors, we develop a post-tool execution reflection component that combines large language model (LLM)-based reflection with domain-specific retrieval-augmented generation (RAG) using documents describing both the specific tool being called and troubleshooting documents related to the tool. For this paper, we focus on the use case of the kubectl command line tool to manage Kubernetes, a platform for orchestrating cluster applications. Through a larger empirical study and a smaller manual evaluation, we find that our RAG-based reflection will repair kubectl commands such that they are both more likely to successfully execute (pass rate) for 55% of our models evaluated and 36% more likely to correctly answer the user query on average. We find that troubleshooting documents improve pass rate compared to official documentation by an average of 10%.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。