用结构化接口提升大模型工具调用可靠性，但语义错误仍难解决。
Schema First Tool APIs for LLM Agents: A Controlled Study of Tool Misuse, Recovery, and Budgeted Performance
- 对比自由文档、JSON Schema和带诊断的Schema三种接口设计
- 结构化接口降低接口误用率，但任务成功率仍为零
- 适合关注本地模型工具调用鲁棒性的研究者参考
工具使用已成为现代大模型代理的核心，但接口设计常未作为独立变量研究。本文探究基于模式的工具契约与结构化验证诊断是否能在严格交互预算下提升可靠性。我们评估三种保持相同工具语义与信息内容的条件：自由格式文档、JSON Schema规范、以及带结构化诊断的JSON Schema。在包含日志、指标、配置与代码库任务的确定性软件工程沙箱中，进行完全交叉的初步实验，涵盖一个开源本地模型、三个随机种子、三种接口条件和四个预算设置。报告了任务成功率、接口误用、执行失败、语义误用、恢复行为与开销等指标。实验发现，各条件下成功率均为零，但采用模式的接口能减少接口误用，无法缓解语义误用。结果表明，接口形式化可增强契约遵守，但在受限本地推理下，语义动作质量与超时敏感任务仍是主要瓶颈。
原文摘要 · Abstract (English)
Tool use has become central to modern LLM agents, yet interface design is rarely isolated as an experimental variable. This paper studies whether schema based tool contracts and structured validation diagnostics improve reliability under strict interaction budgets. We evaluate three conditions that preserve identical tool semantics and information content: free form documentation, JSON Schema specifications, and JSON Schema with structured diagnostics. We implement a deterministic software engineering sandbox with logs, metrics, configurations, and repository tasks, and evaluate a fully crossed pilot with one open local model, three seeds, three interface conditions, and four budgets. We report end task success, interface misuse, execution failures, semantic misuse, recovery behavior, and overhead. In this pilot, success remains zero across conditions, while schema conditions reduce interface misuse but not semantic misuse. The evidence supports a precise interpretation that interface formalization improves contract adherence, but semantic action quality and timeout sensitive tasks remain dominant bottlenecks under constrained local inference.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。