评测大模型用工具完成任务的全过程，发现其常见错误和性能瓶颈。
TRAJECT-Bench:A Trajectory-Aware Benchmark for Evaluating Agentic Tool Use
- 设计了可执行的工具与真实接口结合的任务，追踪工具调用轨迹。
- 在复杂任务中发现模型易混淆相似工具、忽略参数配置等问题。
- 适合研究智能体工具使用能力的开发者与评估者参考。
基于大语言模型（LLM）的智能体越来越多地依赖工具完成现实任务。现有评估多关注最终答案，却忽视了工具使用过程中的细节轨迹——如工具选择、参数设置和调用顺序是否正确。我们提出 TRAJECT-Bench，一个面向轨迹感知的基准评测体系，通过多样化任务和细粒度指标全面评估 LLM 的工具使用能力。该基准将高保真可执行工具与基于生产级 API 的实际任务配对，并生成具有不同广度（并行调用）和深度（依赖链）的工具调用轨迹。除最终准确率外，还报告工具选择、参数正确性及依赖/顺序满足度等轨迹级诊断信息。分析揭示了相似工具混淆、参数盲选等失败模式，并发现工具多样性与轨迹长度带来的缩放行为，其中从短到中等长度轨迹的过渡存在明显瓶颈，为提升模型工具使用能力提供可操作指导。
原文摘要 · Abstract (English)
Large language model (LLM)-based agents increasingly rely on tool use to complete real-world tasks. While existing works evaluate the LLMs' tool use capability, they largely focus on the final answers yet overlook the detailed tool usage trajectory, i.e., whether tools are selected, parameterized, and ordered correctly. We introduce TRAJECT-Bench, a trajectory-aware benchmark to comprehensively evaluate LLMs' tool use capability through diverse tasks with fine-grained evaluation metrics. TRAJECT-Bench pairs high-fidelity, executable tools across practical domains with tasks grounded in production-style APIs, and synthesizes trajectories that vary in breadth (parallel calls) and depth (interdependent chains). Besides final accuracy, TRAJECT-Bench also reports trajectory-level diagnostics, including tool selection and argument correctness, and dependency/order satisfaction. Analyses reveal failure modes such as similar tool confusion and parameter-blind selection, and scaling behavior with tool diversity and trajectory length where the bottleneck of transiting from short to mid-length trajectories is revealed, offering actionable guidance for LLMs' tool use.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。