针对大模型工具文档的动态优化框架,提升任务成功率。
DOCSCHISEL: Adaptive Tool Documentation Optimization Framework for LLM Agents

- 基于失败执行轨迹分析,动态调整工具文档内容。
- 任务成功率提升95.89%,优于现有方法75.15%以上。
- 适合需高效调用外部工具的智能体系统开发者使用。
大型语言模型(LLMs)越来越依赖外部工具完成复杂现实任务,工具文档成为其关键信息来源。现有研究多关注提升模型工具使用能力,而将文档视为固定输入。尽管近期有工作尝试通过重写或压缩优化文档,但尚不清楚文档信息如何影响不同场景下的代理性能。为此,我们开展大规模实证研究,发现现有工具文档在信息字段上存在显著差异,且其有效性高度依赖任务领域、模型主干与代理范式,表明单一文档难以跨场景通用。受此启发,我们提出 DocsChisel 框架:通过分析目标代理的失败执行轨迹,识别文档相关问题,并迭代增删与精炼各工具的信息字段。在 EasyTool 与 DRAFT 两个先进基线上的实验表明,该框架在原始文档基础上使任务成功率提升 95.89%,平均优于现有基线 75.15%,且优化耗时与令牌开销极低。
原文摘要 · Abstract (English)
Large language models (LLMs) increasingly rely on external tools to accomplish complex real-world tasks, making tool documentation a critical grounding resource for LLM agents. Existing studies mainly focus on improving the tool-use capabilities of LLM agents, while largely treating tool documentation as a fixed input. Although several recent works attempt to optimize tool documentation through rewriting or compression, little is known about how the information contained in tool documentation affects agent performance across different settings. To bridge this gap, we conduct a large-scale empirical study on tool documentation for LLM agents. Our study reveals substantial heterogeneity in the information fields provided by existing tool documentation. Moreover, the effectiveness of different information fields is highly dependent on the task domain, LLM backbone, and agent paradigm, indicating that no fixed tool documentation can consistently generalize across diverse agent settings. Motivated by these findings, we propose DocsChisel, an adaptive tool documentation optimization framework for LLM agents. DocsChisel analyzes failed execution traces of a target LLM agent to identify documentation-related issues, and iteratively optimizes tool documentation by adding, removing, and refining information fields for each tool. We evaluate DocsChisel against two state-of-the-art baselines, i.e., EasyTool and DRAFT. Experimental results show that DocsChisel improves the task success rate of LLM agents by 95.89% over the original tool documentation and by 75.15%, on average, over existing baselines, while incurring limited optimization time and token overhead
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。