新基准揭示大模型导航弱于工具使用，1400个复杂路径任务中仅37%完成
The Amazing Agent Race: Strong Tool Users, Weak Navigators
- 设计带分叉合并的图结构任务，模拟真实多步工具调用场景
- 1400个任务中最佳模型准确率仅37.2%，导航错误占比超四成
- 揭示线性基准忽略导航缺陷，适合评估复杂推理与规划能力
现有大模型工具使用基准大多为线性流程：对六个基准分析显示，55%至100%的任务仅为2至5步的简单链式操作。我们提出《The Amazing Agent Race》（AAR）基准，包含具有分叉-合并结构的有向无环图（DAG）谜题（称为“赛道”），支持多步工具链执行。发布共1400个实例，分为两种变体：顺序型（800条赛道）和组合型（600个DAG赛道）。任务要求代理在维基百科中导航、执行多步工具链，并整合结果生成可验证答案。赛道基于维基百科种子程序生成，分为四个难度等级，采用实时API验证。引入三项互补指标：终点准确率、中途节点访问率、障碍完成率，分别诊断导航、工具使用与算术错误。在1400条赛道上评估三个代理框架，最优模型准确率仅为37.2%。导航错误占所有试验的27%至52%，而工具使用错误低于17%；代理架构影响与模型规模相当（Claude Code以6倍更少的令牌量达到Codex CLI的37%准确率）。组合结构揭示：代理失败并非因调用工具，而是导航至正确页面的能力不足，这一盲点在线性基准中无法察觉。
原文摘要 · Abstract (English)
Existing tool-use benchmarks for LLM agents are overwhelmingly linear: our analysis of six benchmarks shows 55 to 100% of instances are simple chains of 2 to 5 steps. We introduce The Amazing Agent Race (AAR), a benchmark featuring directed acyclic graph (DAG) puzzles (or "legs") with fork-merge tool chains. We release 1,400 instances across two variants: sequential (800 legs) and compositional (600 DAG legs). Agents must navigate Wikipedia, execute multi-step tool chains, and aggregate results into a verifiable answer. Legs are procedurally generated from Wikipedia seeds across four difficulty levels with live-API validation. Three complementary metrics (finish-line accuracy, pit-stop visit rate, and roadblock completion rate) separately diagnose navigation, tool-use, and arithmetic failures. Evaluating three agent frameworks on 1,400 legs, the best achieves only 37.2% accuracy. Navigation errors dominate (27 to 52% of trials) while tool-use errors remain below 17%, and agent architecture matters as much as model scale (Claude Code matches Codex CLI at 37% with 6x fewer tokens). The compositional structure of AAR reveals that agents fail not at calling tools but at navigating to the right pages, a blind spot invisible to linear benchmarks. The project page can be accessed at: https://minnesotanlp.github.io/the-amazing-agent-race
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。