评测大模型在跨API与文档推理中的多跳能力，发现其性能随复杂度激增而严重下降。
VAKRA: Evaluating Multi-Hop Reasoning Across APIs and Retrieval Under Tool-Use Policies
- 构建8000+接口覆盖62个领域的多跳推理基准VAKRA
- 单跳任务准确率70.4%，多跳任务降至50-51%，深度增加超50%性能衰减
- 语言理解短板导致实体消歧和跨源对齐失败，而非调用技术问题
企业部署的智能体需在结构化API与文档集合间进行推理，但现有基准仅孤立评估。我们提出VAKRA（评估API与知识检索代理），包含超过8,000个可执行API，覆盖62个领域，任务涵盖三种难度递增场景：多样式API交互、结构化API上的多跳推理，以及受自然语言工具使用策略约束的多源推理。正确性通过重执行预测的工具调用并对比实时API验证，支持多种有效路径。采用固定ReAct框架隔离模型能力与代理架构影响，评估前沿及开源权重模型，结果显示即使最佳模型在单跳端点任务中仅达70.4%准确率，复合型API任务降至50–51%；推理深度增加时性能下降超50%，策略约束问题暴露严重缺陷（不可回答查询低至2.4%）。轨迹分析表明失败集中于语言中介推理环节——实体消歧与跨源对齐，而非工具调用机制本身。代码开源：https://github.com/IBM/VAKRA，数据集：https://huggingface.co/datasets/ibm-research/VAKRA
原文摘要 · Abstract (English)
Agents deployed in enterprise settings must reason across structured APIs and document collections, yet existing benchmarks evaluate these capabilities in isolation. We introduce VAKRA (e\textbf{V}aluating \textbf{A}PI and \textbf{K}nowledge \textbf{R}etrieval \textbf{A}gents), a benchmark of over $8{,}000$ executable APIs across $62$ domains with tasks spanning three settings of increasing difficulty: diverse API interaction styles, multi-hop reasoning over structured APIs, and multi-source reasoning with natural-language tool-use policy constraints. Correctness is verified by re-executing predicted tool calls against live APIs, accommodating multiple valid paths. Using a fixed ReAct harness to isolate model capabilities from agent architecture, we evaluate frontier and open-weight models and find that even the best model achieves only 70.4\% on single-hop endpoint-style tasks and drops to 50--51\% on compositional APIs; performance degrades by over 50\% as reasoning depth increases, and policy-constrained questions expose severe failures (as low as 2.4\% on unanswerable queries). Trace analysis shows failures concentrate at language-mediated reasoning - entity disambiguation, cross-source grounding, rather than tool invocation mechanics. Code is available https://github.com/IBM/VAKRA. Dataset is available https://huggingface.co/datasets/ibm-research/VAKRA
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。