通过目标-计划-行动对齐框架,系统评估智能体失败原因并定位问题。
What Is Your Agent's GPA? A Framework for Evaluating Agent Goal-Plan-Action Alignment
- 用大模型分项判断智能体的目标、计划与执行是否一致。
- 在三个测试集上识别出95%的人工标注错误,定位准确率达86%。
- 自动优化评价标准,比人工提示覆盖更多错误,适合开发者调试智能体。
我们提出Agent GPA(目标-计划-行动)框架,核心观点是智能体的关键失败源于目标设定、计划制定与行动执行之间的不一致。通过一套分解式的大语言模型评判器,量化评估三者对齐程度。为实现跨架构与数据集的可扩展性,采用前沿自动化提示优化技术生成领域特定的评估标准。在三个基准上验证:多智能体研究环境(TRAIL/GAIA)、单个编码智能体(TRAIL/SWE-bench)以及企业级数据智能体(Snowflake Intelligence)。TRAIL/GAIA的大量实验表明,该框架能识别95%的人工标注错误,86%的错误可被准确定位,且与人类评价高度一致。关键的是,相比人工提示方法,本框架在公开数据集上实现了76%至86%的错误覆盖率。此外,通过进化式编码智能体迭代优化评价标准,裁判一致性提升最高达38%。Agent GPA提供了一种严谨且通用的智能体针对性评估范式。
原文摘要 · Abstract (English)
We introduce the Agent GPA (Goal-Plan-Action) framework, driven by the fundamental insight that critical agent failures emerge at the intersections of setting goals, devising plans, and executing actions. We operationalize the framework with a factorized suite of LLM judges designed to measure distinct elements of Goal-Plan-Act alignment. To make this methodology scalable and generalizable across diverse agent architectures and datasets, we use state-of-the-art automated prompt optimization techniques to systematically generate domain-specific evaluation criteria. We validate this approach across three benchmarks: a multi-agent research setting (TRAIL/GAIA), a single coding agent setting (TRAIL/SWE-bench), and a private, enterprise data-agent setting (Snowflake Intelligence). Extensive evaluation on TRAIL/GAIA demonstrates the core validity of the framework, which identifies a broad range of agent failures (95% of human-annotated errors), localizes errors to enable targeted debugging (86% of human-annotated errors), and exhibits strong agreement with human evaluators. Crucially, by applying our automated methodology to both public datasets, we demonstrate that our GPA judges generally achieve the highest error coverage (ranging from 76% to 86%) in comparison to manual prompting approaches. We also leverage an evolutionary coding agent to improve judge consistency by up to 38% through iterative refinement of evaluation rubrics. Overall, Agent GPA provides a rigorous and generalizable paradigm for targeted agent evaluation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。