用智能体评估智能体，实现全过程动态反馈。
Agent-as-a-Judge: Evaluate Agents with Agents
- 用智能体作为评判者，全程评估任务执行过程。
- 在55个真实开发任务上，评分可靠性接近人工基准。
- 适合研究智能体自我优化与自动评估的学者。
当前评估方法难以有效衡量智能体系统。现有方法要么只关注最终结果，忽略执行过程；要么依赖大量人工标注。为此，我们提出Agent-as-a-Judge框架，让智能体评价智能体。该框架扩展了LLM-as-a-Judge，引入中间反馈机制，支持全过程评估。我们在代码生成任务上应用该框架，并构建了包含55个真实自动化开发任务的新基准DevAI，涵盖总计365个层级用户需求的精细标注。对三种主流智能体系统进行评估发现，Agent-as-a-Judge显著优于LLM-as-a-Judge，且可靠性与人工评估相当。我们认为，该框架为智能体系统的动态、可扩展自进化提供了关键的高质量奖励信号。
原文摘要 · Abstract (English)
Contemporary evaluation techniques are inadequate for agentic systems. These approaches either focus exclusively on final outcomes -- ignoring the step-by-step nature of agentic systems, or require excessive manual labour. To address this, we introduce the Agent-as-a-Judge framework, wherein agentic systems are used to evaluate agentic systems. This is an organic extension of the LLM-as-a-Judge framework, incorporating agentic features that enable intermediate feedback for the entire task-solving process. We apply the Agent-as-a-Judge to the task of code generation. To overcome issues with existing benchmarks and provide a proof-of-concept testbed for Agent-as-a-Judge, we present DevAI, a new benchmark of 55 realistic automated AI development tasks. It includes rich manual annotations, like a total of 365 hierarchical user requirements. We benchmark three of the popular agentic systems using Agent-as-a-Judge and find it dramatically outperforms LLM-as-a-Judge and is as reliable as our human evaluation baseline. Altogether, we believe that Agent-as-a-Judge marks a concrete step forward for modern agentic systems -- by providing rich and reliable reward signals necessary for dynamic and scalable self-improvement.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。