提升界面感知精度与解析能力,实现端到端高精度界面元素定位。
SparkUI-Parser: Enhancing GUI Perception with Robust Grounding and Parsing
- 采用连续坐标建模替代离散输出,提升定位精度与推理速度。
- 引入拒绝机制识别虚假元素,降低误报率,增强鲁棒性。
- 构建新基准ScreenParse,全面评估界面结构理解能力,适合研究者使用。
现有用于界面感知的多模态大模型虽取得进展,但仍面临两大挑战:一是基于文本自回归机制建模离散坐标,导致定位精度低且推理慢;二是仅能定位预定义元素,无法解析完整界面,限制下游任务应用。为此,我们提出SparkUI-Parser,一种端到端框架,同时实现高精度定位与细粒度界面解析。通过在预训练多模态大模型基础上引入令牌路由器与坐标解码器,以连续方式建模坐标,克服离散输出和逐标记生成的局限,显著提升准确率与速度。为进一步增强鲁棒性,设计基于改进匈牙利匹配的拒绝机制,可识别并剔除不存在元素,减少误报。此外,提出ScreenParse基准,系统评估模型在多样化场景下的结构感知能力。大量实验表明,本方法在ScreenSpot、ScreenSpot-v2、CAGUI-Grounding及ScreenParse等多个基准上持续优于当前最优(SOTA)方法。资源已开源:https://github.com/antgroup/SparkUI-Parser。
原文摘要 · Abstract (English)
The existing Multimodal Large Language Models (MLLMs) for GUI perception have made great progress. However, the following challenges still exist in prior methods: 1) They model discrete coordinates based on text autoregressive mechanism, which results in lower grounding accuracy and slower inference speed. 2) They can only locate predefined sets of elements and are not capable of parsing the entire interface, which hampers the broad application and support for downstream tasks. To address the above issues, we propose SparkUI-Parser, a novel end-to-end framework where higher localization precision and fine-grained parsing capability of the entire interface are simultaneously achieved. Specifically, instead of using probability-based discrete modeling, we perform continuous modeling of coordinates based on a pre-trained Multimodal Large Language Model (MLLM) with an additional token router and coordinate decoder. This effectively mitigates the limitations inherent in the discrete output characteristics and the token-by-token generation process of MLLMs, consequently boosting both the accuracy and the inference speed. To further enhance robustness, a rejection mechanism based on a modified Hungarian matching algorithm is introduced, which empowers the model to identify and reject non-existent elements, thereby reducing false positives. Moreover, we present ScreenParse, a rigorously constructed benchmark to systematically assess structural perception capabilities of GUI models across diverse scenarios. Extensive experiments demonstrate that our approach consistently outperforms SOTA methods on ScreenSpot, ScreenSpot-v2, CAGUI-Grounding and ScreenParse benchmarks. The resources are available at https://github.com/antgroup/SparkUI-Parser.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。