用真实执行验证替代语义匹配，提升大模型工具选择准确性
GRETEL: A Goal-driven Retrieval and Execution-based Trial Framework for LLM Tool Selection Enhancing
- 通过沙箱环境试运行候选工具，验证其实际可用性
- 在ToolBench上工具成功率从69.0%提升至82.6%
- 适合需要可靠工具调用的智能体开发场景
尽管大型语言模型能力显著进步，基于代理系统的工具检索仍受限于依赖语义相似性，难以捕捉功能可行性。现有方法常选出文本相关但功能无效的工具，因参数不匹配、认证失败或执行限制等问题——我们称此为语义-功能鸿沟。为此提出GRETEL，通过系统性实证验证解决该问题。GRETEL采用代理式工作流，对语义检索到的候选工具进行沙箱内的计划-执行-评估循环，生成基于执行结果的证据，以区分真正可用与仅描述匹配的工具。在ToolBench基准上的全面评估显示，各项指标均有显著提升：通过率（10）从0.690增至0.826，召回率（10）从0.841升至0.867，NDCG（10）由0.807提高到0.857。结果表明，基于执行的验证比单纯语义相似性更可靠，可增强实际应用中智能体的鲁棒性。
原文摘要 · Abstract (English)
Despite remarkable advances in Large Language Model capabilities, tool retrieval for agent-based systems remains fundamentally limited by reliance on semantic similarity, which fails to capture functional viability. Current methods often retrieve textually relevant but functionally inoperative tools due to parameter mismatches, authentication failures, and execution constraints--a phenomenon we term the semantic-functional gap. We introduce GRETEL, to address this gap through systematic empirical validation. GRETEL implements an agentic workflow that processes semantically retrieved candidates through sandboxed plan-execute-evaluate cycles, generating execution-grounded evidence to distinguish truly functional tools from merely descriptive matches. Our comprehensive evaluation on the ToolBench benchmark demonstrates substantial improvements across all metrics: Pass Rate (at 10) increases from 0.690 to 0.826, Recall (at 10) improves from 0.841 to 0.867, and NDCG (at 10) rises from 0.807 to 0.857.. These results establish that execution-based validation provides a more reliable foundation for tool selection than semantic similarity alone, enabling more robust agent performance in real-world applications.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。