构建跨平台GUI接地基准,支持多层级评估真实应用。
VenusBench-GD: A Comprehensive Multi-Platform GUI Benchmark for Diverse Grounding Tasks
- 构建覆盖多平台的大型双语标注数据集,涵盖丰富界面元素。
- 通用多模态模型在基础任务上已超越专用模型,但高级任务仍依赖专业模型。
- 提出分层任务分类法,适配不同阶段的GUI智能体研发需求。
GUI接地是构建高效GUI智能体的关键。现有基准存在数据量不足、领域覆盖窄或过度聚焦单一平台、需高度专业领域知识等问题。本文提出VenusBench-GD,一个全面的多平台双语GUI接地基准,支持真实应用场景的分层评估。主要贡献包括:(i) 构建大规模跨平台基准,覆盖广泛的应用场景、多样化的UI元素和丰富的标注数据;(ii) 建立高质量的数据构建流程,标注准确率高于现有基准;(iii) 提出分层任务分类体系,将接地任务分为基础与高级两类,包含六种子任务,从不同角度评估模型性能。实验发现:通用多模态模型在基础任务上已达或超过专用模型表现;而高级任务仍由专用模型主导,但存在严重过拟合和鲁棒性差问题。结果凸显全面、多层级评估框架的必要性。
原文摘要 · Abstract (English)
GUI grounding is a critical component in building capable GUI agents. However, existing grounding benchmarks suffer from significant limitations: they either provide insufficient data volume and narrow domain coverage, or focus excessively on a single platform and require highly specialized domain knowledge. In this work, we present VenusBench-GD, a comprehensive, bilingual benchmark for GUI grounding that spans multiple platforms, enabling hierarchical evaluation for real-word applications. VenusBench-GD contributes as follows: (i) we introduce a large-scale, cross-platform benchmark with extensive coverage of applications, diverse UI elements, and rich annotated data, (ii) we establish a high-quality data construction pipeline for grounding tasks, achieving higher annotation accuracy than existing benchmarks, and (iii) we extend the scope of element grounding by proposing a hierarchical task taxonomy that divides grounding into basic and advanced categories, encompassing six distinct subtasks designed to evaluate models from complementary perspectives. Our experimental findings reveal critical insights: general-purpose multimodal models now match or even surpass specialized GUI models on basic grounding tasks. In contrast, advanced tasks, still favor GUI-specialized models, though they exhibit significant overfitting and poor robustness. These results underscore the necessity of comprehensive, multi-tiered evaluation frameworks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。