arXiv:2605.17439cs.SEcs.AI2026-05被引 1

通过复用失败轨迹诊断软件缺陷,提升GUI Agent评估可靠性。

DiagEval: Trajectory-Conditioned Diagnosis for Reliable Software Evaluation with GUI Agents

论文配图:DiagEval: Trajectory-Conditioned Diagnosis for Reliable Software Evaluation with GUI Agents
图 1 · 摘自论文原文
  • 基于失败轨迹设计诊断探针,精准定位问题来源
  • 在真实数据集上使误判率降低45.6%-62.1%
  • 适合需要高可信度软件评估的开发者与研究者

评估大模型生成的交互式软件需结合执行与静态分析。难点在于正确性是隐式UI状态转移图上的全局可达属性,而GUI评估器仅观测单一执行轨迹。一次失败仅排除一条路径,难以区分是评估器自身错误还是真实软件缺陷。本文提出DiagEval,一种轨迹条件化的诊断评估协议。不盲目重试,而是复用失败轨迹,选择针对性诊断探针,并聚合结果生成内部归因信号。该方法不重构图结构或估计后验概率。在WebDevJudge-Unit和RealDevBench上测试,对假负例,可恢复45.6%-62.1%被误判为缺陷的失败案例,相对基线提升34.4%-160.6%。全集评估中,准确率从69.9%提升至78.3%(WebDevJudge-Unit),从65.0%提升至81.6%(RealDevBench)。结果表明,可靠评估不仅需更强执行能力,还需主动诊断以区分评估误差与真实缺陷。代码已开源。

原文摘要 · Abstract (English)

Evaluating LLM-generated interactive software requires execution in addition to static analysis. The key difficulty is that correctness is a graph-level reachable property over latent UI state-transition graphs, whereas a GUI evaluator observes only a single execution trajectory. A failed rollout therefore rules out only one realized path, leaving failure attribution ambiguous between evaluator-side execution error and genuine software defect. We present DiagEval, a trajectory-conditioned diagnostic evaluation protocol for post-failure GUI-agent evaluation of interactive software. Rather than blindly retrying from scratch, DiagEval reuses the failed trajectory to choose targeted diagnostic probes and aggregates their outcomes into an internal attribution signal. The latent-graph view motivates the diagnostic problem; DiagEval does not reconstruct the graph or estimate calibrated posterior probabilities. We evaluate DiagEval on WebDevJudge-Unit and RealDevBench across multiple GUI-agent evaluators and LLM backbones. On false-negative cases, DiagEval recovers 45.6-62.1% of failures that were initially misattributed to software defects, outperforming retry-based baselines with 34.4-160.6% relative gains. On the full evaluation sets, this recovery improves accuracy from 69.9% to 78.3% on WebDevJudge-Unit and from 65.0% to 81.6% on RealDevBench. These results suggest that reliable GUI-agent evaluation requires not only stronger execution, but also active failure diagnosis to disambiguate evaluator-side errors from genuine software defects. Our code is available at https://github.com/scutGit/DiagEval.

GUI评估故障诊断大模型评估软件验证

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。