用智能体当裁判评估大模型行为，更可靠。
AJ-Bench: Benchmarking Agent-as-a-Judge for Environment-Aware Evaluation
- 让智能体主动与环境交互获取证据，实现动态验证。
- 在155个任务上优于传统语言模型裁判，但仍有明显短板。
- 适合研究智能体评估、自主系统验证的学者使用。
随着基于大语言模型的智能体在强化学习中规模不断扩大，如何在复杂环境中可靠验证其行为变得日益困难。现有方法依赖规则引擎或语言模型作为裁判，难以跨领域泛化。本文提出一种新型方法：让智能体作为裁判，主动与环境和工具交互以获取可验证证据。为此，我们构建了AJ-Bench基准，涵盖搜索、数据系统和图形界面三个领域，包含155个任务和516条标注轨迹，全面评估裁判智能体在信息获取、状态验证和过程验证方面的能力。实验表明，该方法在多数任务上持续优于基于语言模型的裁判基线，但仍暴露出显著的开放挑战。相关数据与代码已开源：https://aj-bench.github.io/。
原文摘要 · Abstract (English)
As reinforcement learning continues to scale the training of large language model-based agents, reliably verifying agent behaviors in complex environments has become increasingly challenging. Existing approaches rely on rule-based verifiers or LLM-as-a-Judge models, which struggle to generalize beyond narrow domains. Agent-as-a-Judge addresses this limitation by actively interacting with environments and tools to acquire verifiable evidence, yet its capabilities remain underexplored. We introduce a benchmark AJ-Bench to systematically evaluate Agent-as-a-Judge across three domains-search, data systems, and graphical user interfaces-comprising 155 tasks and 516 annotated trajectories. The benchmark comprehensively assesses judge agents' abilities in information acquisition, state verification, and process verification. Experiments demonstrate consistent performance gains over LLM-as-a-Judge baselines, while also revealing substantial open challenges in agent-based verification. Our data and code are available at https://aj-bench.github.io/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。