A2E可自动评估智能体框架性能，发现不同组合表现差异大。
$A^2E$ : An End-to-End Agent Auditing Engine
- 用新协议ATP快速接入多种框架，自动采集标准化执行日志。
- 多维度指标显示框架在效率、工具使用、规划等能力上差异显著。
- 适合研究者和开发者系统评估模型与框架的协同表现。
随着大语言模型的快速发展，智能体框架已成为跨领域部署智能体的关键基础设施。快速演进的框架生态也使得严格的性能评估变得愈发重要。然而，构建端到端、系统化且全面的评估流水线仍面临重大挑战。为此，我们提出A2E（Agent Auditing Engine），一个面向智能体框架的端到端评估引擎。A2E采用新提出的智能体任务协议（ATP），实现评估任务与不同框架的快速集成；通过自动注入的监控器，在实验过程中捕获并生成标准化执行轨迹。在评估阶段，A2E利用一套多维指标对框架能力进行系统性分析，相比仅关注正确性，这些指标能更精细地刻画框架在执行效率、工具使用、任务规划及错误恢复等方面的差异。基于A2E的实验表明，模型与框架的组合在不同类型任务中表现出显著性能差异，且不存在在所有任务中均占优的单一组合。这些发现不仅证明了系统化评估的必要性，也为模型与框架的协同演化提供了实用指导。代码已开源：https://github.com/datamllab/A2E。
原文摘要 · Abstract (English)
With the rapid advancement of large language models (LLMs), harnesses have become essential infrastructure for deploying agents across a wide range of domains. The fast-evolving harness ecosystem has also made rigorous capability evaluation increasingly important. However, efficiently building an end-to-end, systematic, and comprehensive evaluation pipeline remains a significant challenge. To address this challenge, we introduce $A^2E$ (Agent Auditing Engine), an end-to-end evaluation engine designed for agent harnesses. $A^2E$ leverages our newly proposed Agent Task Protocol (ATP) to enable the rapid integration of evaluation tasks with different harnesses. Through an automatically instrumented Monitor, it captures and generates standardized execution traces during experiments. In the Evaluation stage, $A^2E$ systematically assesses harness capabilities using a suite of multidimensional metrics. Compared with correctness alone, these metrics provide a more fine-grained characterization of differences among harnesses in execution efficiency, tool use, task planning, and error recovery. Experiments conducted with $A^2E$ further reveal that model-harness combinations exhibit substantial performance variation across different types of tasks, and that no single combination consistently outperforms all others across every task. These findings not only demonstrate the necessity of systematic evaluation but also provide useful guidance for the co-evolving of models and harnesses. Our code is available at https://github.com/datamllab/A2E.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。