通过有向无环图追踪错误传播，精准定位智能体工作流中的中间故障。
AgentEval: DAG-Structured Step-Level Evaluation for Agentic Workflows with Error Propagation Tracking
- 将智能体执行过程建模为带依赖关系的有向无环图，节点携带量化质量指标。
- 相比端到端评估，故障检出率提升2.17倍，根因定位准确率达72%。
- 可直接用于线上集成，4个月实测显著缩短故障排查时间并降低失败率。
能够将推理、工具使用与合成串联成多步骤工作流的智能体系统正进入生产环境，但现有评估方法如端到端结果检查和临时轨迹分析，会系统性掩盖主导真实错误预算的中间阶段失败。本文提出AgentEval，将智能体执行过程形式化为评估有向无环图（DAG），每个节点包含由校准过的LLM判官（GPT-4o）评估的类型化质量指标，依据三级共21个子类别的故障分类体系进行归类，并链接上游依赖关系以实现自动根因追溯。消融实验表明，仅依赖关系建模一项就使故障检出召回率提升22个百分点，根因定位准确率提升34个百分点，优于相同评判标准下的扁平化步骤评估。在三个生产级工作流（450个测试用例，两个智能体模型族，主要为顺序架构，非DAG轨迹占比12%）上，AgentEval的故障检出召回率为0.89，是端到端评估（0.41）的2.17倍，与人类专家的一致性达Cohen's kappa = 0.84，根因定位准确率为72%，接近人类天花板（81%）。在tau-bench和SWE-bench数据集上的跨系统评估验证了其可迁移性（故障检出召回率≥0.78），无需调整分类体系或评分标准。为期4个月的试点中，18名工程师通过集成至CI/CD的回归测试发现23个发布前缺陷，根因定位时间从平均4.2小时缩短至22分钟，两个工作流的失败率明显下降。
原文摘要 · Abstract (English)
Agentic systems that chain reasoning, tool use, and synthesis into multi-step workflows are entering production, yet prevailing evaluation practices like end-to-end outcome checks and ad-hoc trace inspection systematically mask the intermediate failures that dominate real-world error budgets. We present AgentEval, a framework that formalizes agent executions as evaluation directed acyclic graphs (DAGs), where each node carries typed quality metrics assessed by a calibrated LLM judge (GPT-4o), classified through a hierarchical failure taxonomy (3 levels, 21 subcategories), and linked to upstream dependencies for automated root cause attribution. An ablation study isolates the impact of DAG-based dependency modeling: it alone contributes +22 percentage points to failure detection recall and +34 pp to root cause accuracy over flat step-level evaluation with identical judges and rubrics. Across three production workflows (450 test cases, two agent model families, predominantly sequential architectures with a 12% non-DAG trace rate), AgentEval achieves 2.17x higher failure detection recall than end-to-end evaluation (0.89 vs. 0.41), Cohen's kappa = 0.84 agreement with human experts, and 72% root cause accuracy against an 81% human ceiling. Cross-system evaluation on tau-bench and SWE-bench traces confirms transferability (failure detection recall >= 0.78) without taxonomy or rubric modification. A 4-month pilot with 18 engineers detected 23 pre-release regressions through CI/CD-integrated regression testing, reducing median root-cause identification time from 4.2 hours to 22 minutes and driving measurable failure rate reductions in two workflows.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。