让大模型通过交互学习模糊工具的使用，提升真实场景任务完成率。
OpaqueToolsBench: Learning Nuances of Tool Behavior Through Interaction
- 通过观察执行反馈迭代优化工具文档，改进模型对模糊工具的理解。
- 在三类任务环境中，新方法性能超越现有方案，尤其在复杂场景更优。
- 测试时探索效率高，消耗令牌数仅为最优基线的1/7至1/3。
工具调用是大型语言模型代理完成现实任务的关键。然而，现有基准大多假设工具简单且文档清晰，而真实世界中的工具（如通用搜索API）往往模糊不清，缺乏明确的最佳实践或失败模式。本研究构建了OpaqueToolsBench基准，包含三类面向任务的环境：通用函数调用、交互式国际象棋对弈和长轨迹代理搜索。每个环境均提供说明不全的工具，要求模型通过交互学习有效使用。实验表明，现有自动文档生成方法在模糊工具上成本高且不可靠。为此，提出ToolObserver框架，通过分析工具调用轨迹的执行反馈，迭代改进文档。该方法在多个数据集上表现更优，即使在较难设置下也显著领先；在测试阶段工具探索场景中，效率更高，总令牌消耗为最优基线的3.5至7.5倍减少。
原文摘要 · Abstract (English)
Tool-calling is essential for Large Language Model (LLM) agents to complete real-world tasks. While most existing benchmarks assume simple, perfectly documented tools, real-world tools (e.g., general "search" APIs) are often opaque, lacking clear best practices or failure modes. Can LLM agents improve their performance in environments with opaque tools by interacting and subsequently improving documentation? To study this, we create OpaqueToolsBench, a benchmark consisting of three distinct task-oriented environments: general function calling, interactive chess playing, and long-trajectory agentic search. Each environment provides underspecified tools that models must learn to use effectively to complete the task. Results on OpaqueToolsBench suggest existing methods for automatically documenting tools are expensive and unreliable when tools are opaque. To address this, we propose a simple framework, ToolObserver, that iteratively refines tool documentation by observing execution feedback from tool-calling trajectories. Our approach outperforms existing methods on OpaqueToolsBench across datasets, even in relatively hard settings. Furthermore, for test-time tool exploration settings, our method is also efficient, consuming 3.5-7.5x fewer total tokens than the best baseline.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。