提出新型评估框架，精准定位AI代理执行失败的环节。
Holistic Evaluation and Failure Diagnosis of AI Agents
- 分层评估：从整体到片段逐级诊断，可处理任意长度执行轨迹。
- 在TRAIL基准上，错误定位准确率提升至基线的3.5倍，联合定位分类达12.5倍。
- 适用于需要高精度故障分析的AI系统研发与调试场景。
AI代理执行复杂多步任务，但现有评估方法存在局限：结果指标仅报告成功或失败，无法解释原因；过程级方法难以将失败类型精确定位到长而结构化的执行轨迹中。本文提出一种整体性代理评估框架，结合自上而下的代理级诊断与自下而上的片段级评估，将分析分解为独立的片段评估，可扩展至任意长度的轨迹，并生成每个判断的片段级解释。在TRAIL基准测试中，该框架在GAIA和SWE-Bench两个数据集上均达到最优表现，相较于最强基线，类别F1最高提升38%，定位准确率最高提升3.5倍，联合定位-分类准确率最高提升12.5倍。按类别分析显示，本框架在多数错误类型上优于其他评估器。值得注意的是，同一前沿模型在本框架内使用时，定位准确率显著高于作为整体判断器直接评估完整轨迹，表明评估方法而非模型能力才是瓶颈。
原文摘要 · Abstract (English)
AI agents execute complex multi-step processes, but current evaluation falls short: outcome metrics report success or failure without explaining why, and process-level approaches struggle to connect failure types to their precise locations within long, structured traces. We present a holistic agent evaluation framework that pairs top-down agent-level diagnosis with bottom-up span-level evaluation, decomposing analysis into independent per-span assessments. This decomposition scales to traces of arbitrary length and produces span-level rationales for each verdict. On the TRAIL benchmark, our framework achieves state-of-the-art results across all metrics on both GAIA and SWE-Bench, with relative gains over the strongest prior baselines of up to 38% on category F1, up to 3.5x on localization accuracy, and up to 12.5x on joint localization-categorization accuracy. Per-category analysis shows our framework leading in more error categories than any other evaluator. Notably, the same frontier model achieves several times higher localization accuracy when used inside our framework than as a monolithic judge over the full trace, showing that evaluation methodology, not model capability, is the bottleneck.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。