用分步推理评估智能体轨迹,兼顾任务完成与操作安全
Automated Trajectory Evaluation for Mobile Agents via Step-Level Consequence Reasoning and Aggregation

- 分步分析每一步的视觉线索与状态变化
- 在AndroidWorld上任务评估F1达0.833,比基线高20%
- 可适配开源闭源模型,适合智能体评测研究者
评估语言引导的移动智能体正从规则驱动转向模型驱动,以实现可扩展的自动化评估。然而,现有整体评估范式一次性处理完整轨迹,导致上下文过载,且主要关注任务完成而忽略操作安全。为此,我们提出CRATE,一种兼容开闭源模型的两阶段视觉语言模型判官框架。CRATE通过分步后果推理机制,在每一步独立提取任务相关的视觉线索,并推断动作条件下的状态变化。生成的步骤级文本证据经轨迹级聚合后,提供基于证据的任务完成评估。在此基础上,进一步扩展为CRATE-S用于操作安全评估。大量实验验证了两者有效性和鲁棒性。基于Qwen2.5-VL-72B-Instruct,CRATE在AndroidWorld上达到0.833的F1分数(优于SPA-Bench 20%),CRATE-S在MobileRisk上获得0.697的F1分数,与基准真值高度一致。代码已公开于https://anonymous.4open.science/r/CRATE-D580。
原文摘要 · Abstract (English)
Evaluating language-guided mobile agents has recently shifted from rule-based to model-based approaches to achieve scalable and automated assessments. However, existing holistic evaluation paradigms process entire trajectories at once, leading to substantial context overload. Moreover, they primarily focus on task completion while overlooking operational safety. To address these limitations, we introduce CRATE, a novel two-stage VLM-as-judge framework for automated mobile agent evaluation that is compatible with both open- and closed-source models. Leveraging a step-level consequence reasoning mechanism, CRATE independently extracts task-relevant visual clues and infers action-conditioned state changes at each step. The resulting step-level textual evidence is then synthesized through trajectory-level aggregation to deliver an evidence-grounded evaluation of task completion. Building upon this evaluation scheme, we further extend CRATE to CRATE-S for operational safety assessment. Extensive experiments validate the effectiveness and robustness of both CRATE and CRATE-S. Powered by Qwen2.5-VL-72B-Instruct, CRATE achieves an F1-score of 0.833 on AndroidWorld (outperforming SPA-Bench by 20%), while CRATE-S reaches an F1-score of 0.697 on MobileRisk, demonstrating strong alignment with benchmark ground truths. Code is available at https://anonymous.4open.science/r/CRATE-D580.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。