用诊断框架拆解大模型智能体行为，揭示成功背后的决策质量。
AgentAtlas: Beyond Outcome Leaderboards for LLM Agents
- 提出六类控制决策分类，系统化标注智能体行为
- 构建轨迹失败词汇表，定位错误根源与影响
- 揭示仅看结果排名的误导性，适合评估设计者与调试者
大语言模型智能体已能在代码库、浏览器、操作系统、日历、文件和工具生态中执行任务，但现有评估常将行为简化为最终任务成功率。AgentAtlas 将评估重构为诊断词汇与审计协议，区分结果成功、控制决策质量与轨迹质量。本文贡献：(i) 六状态控制决策分类（执行 / 提问 / 拒绝 / 停止 / 确认 / 恢复）；(ii) 轨迹失败词汇表，包含主要错误来源与下游影响；(iii) 对十五个智能体基准的 0/1/2 基准覆盖审计；(iv) 在含 1,342 项的合成数据集上，使用八种模型对比分类感知与非感知提示格式的演示研究。该合成实验非公开基准，不用于模型排名。其旨在揭示两种测量风险：标签映射一致性在移除显式标签菜单后显著变化；轴选择可改变模型相对排名。AgentAtlas 旨在帮助基准设计者明确覆盖的行为类型，帮助评估者诊断结果排行榜掩盖的失败模式。
原文摘要 · Abstract (English)
Large language model agents now act on codebases, browsers, operating systems, calendars, files, and tool ecosystems, but their evaluations often collapse behavior into final task success. AgentAtlas reframes agent evaluation as a diagnostic vocabulary and audit protocol for separating outcome success from control-decision quality and trajectory quality. The paper contributes: (i) a six-state control-decision taxonomy (Act / Ask / Refuse / Stop / Confirm / Recover); (ii) a trajectory-failure vocabulary with primary error source and downstream impact; (iii) a 0/1/2 benchmark-coverage audit over fifteen agent benchmarks; and (iv) an illustrative protocol study on a synthetic 1,342-item set evaluated with eight models under taxonomy-aware and taxonomy-blind prompt formats. The synthetic demonstration is not a public benchmark release and should not be read as a definitive model comparison. Instead, it illustrates two measurement risks: mapped label agreement can change substantially when the explicit label menu is removed, and axis choice can change apparent rankings. AgentAtlas is intended to help benchmark designers state what behavior they cover, and to help evaluators diagnose failures that outcome-only leaderboards hide.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。