arXiv:2606.22737cs.AIcs.CL2026-06被引 3

用可复现的测试取代大模型裁判,验证智能体是否真用了正确证据。

GroundEval: A Deterministic Replacement for LLM-as-Judge in Stateful Agent Evaluation

  • 通过配置领域问题,追踪智能体搜索、引用和访问行为路径
  • 同一答案在人工评分中得0.85以上,但实测证据链缺失得分0.000
  • 聚焦沉默、视角、反事实三类漏洞,适合评估需要严格证据链的系统

在让智能体处理真实上下文前,能否证明它使用了正确的证据?GroundEval将此问题转化为对智能体搜索、获取、引用及访问权限的确定性测试。案例研究显示,两个前沿LLM裁判对一个合理响应评分均高于0.85,但轨迹分析揭示:该智能体从未检索其答案所依赖的资料,导致GroundEval得分为0.000。我们提出GroundEval,一种无需裁判的评估框架,针对有时间限制、访问受控且基于证据的智能体进行评测。该框架通过领域配置生成问题,允许智能体自主选择回答方式,并对最终答案与生成轨迹双重评分。基准测试聚焦三种大模型裁判难以检测的失效模式:是否在声称无信息前已检查;是否仅依据角色在当时可访问的证据推理;是否使用了正确的因果机制而非表面合理的推断。这对应三个评估维度:沉默(Silence)、视角(Perspective)与反事实(Counterfactual)。GroundEval能暴露看似合理但基于无效证据路径的答案,并提供逐题结构化诊断,将工具操作与对话层级叙述对齐,使每项得分可追溯而非仅报告。案例研究表明,此类失效模式普遍存在,而最终答案与裁判评价法在设计上无法捕捉。

原文摘要 · Abstract (English)

Before letting an agent operate over real context, can you prove it used the right evidence? GroundEval turns that question into a deterministic test of what the agent searched, fetched, cited, and was permitted to access. In one case study, two frontier LLM judges scored a plausible agent response 0.85 and higher. But the trace told a different story: the agent had never retrieved the artifact its answer depended on, yielding a GroundEval score of 0.000. We introduce GroundEval, a judge-free framework for evaluating agents against grounded, time-bounded, and access-controlled evidence. GroundEval uses a domain configuration to generate questions, lets the agent choose how to answer, and then scores both the final answer and the recorded trajectory that produced it. The benchmark targets three failures that LLM-as-judge evaluation struggles to detect: whether an agent checked before claiming absence, reasoned only from evidence available to the actor at the relevant time, and used the correct causal mechanism rather than a plausible one. These correspond to three tracks: Silence, Perspective, and Counterfactual. GroundEval exposes when plausible answers rest on invalid evidence paths, and produces structured per-question diagnostics that pair tool activity with the agent's turn-level narration, making each score inspectable rather than merely reported. Our case studies suggest this failure mode is common rather than exceptional, one that final-answer and judge-based evaluation cannot detect by construction.

智能体评估证据验证去裁判化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。