分阶段精准定位界面元素，提升复杂场景下的指令理解准确率。
MEGA-GUI: Multi-stage Enhanced Grounding Agents for GUI Elements
- 分两阶段处理：先选区域再精确定位，用专用智能体协同工作。
- 在ScreenSpot-Pro上达73.18%准确率，在OSWorld-G上达68.63%。
- 适合做自动化操作、无障碍访问的开发者参考使用。
图形用户界面（GUI）定位——将自然语言指令映射到屏幕坐标——对自主代理和无障碍技术至关重要。现有系统依赖单一模型或一次性流程，缺乏模块化，在视觉杂乱和指令模糊时表现不佳。我们提出MEGA-GUI，一种多阶段框架，将定位分为粗粒度兴趣区域（ROI）选择与细粒度元素定位，由专用视觉-语言智能体协调。该框架采用双向ROI缩放算法缓解空间稀释问题，并引入上下文感知重写智能体降低语义歧义。分析显示不同视觉尺度下视觉-语言模型具有互补优劣，利用模块化结构可实现比单体方法更稳定的高精度。在视觉密集的ScreenSpot-Pro基准上，MEGA-GUI达到73.18%准确率；在语义复杂的OSWorld-G基准上达到68.63%，优于以往报告结果。代码与接地基准工具包（GBT）已开源：https://github.com/samsungsds-research-papers/mega-gui。
原文摘要 · Abstract (English)
Graphical User Interface (GUI) grounding - the task of mapping natural language instructions to screen coordinates - is essential for autonomous agents and accessibility technologies. Existing systems rely on monolithic models or one-shot pipelines that lack modularity and fail under visual clutter and ambiguous instructions. We introduce MEGA-GUI, a multi-stage framework that separates grounding into coarse Region-of-Interest (ROI) selection and fine-grained element grounding, orchestrated by specialized vision-language agents. MEGA-GUI features a bidirectional ROI zoom algorithm that mitigates spatial dilution and a context-aware rewriting agent that reduces semantic ambiguity. Our analysis reveals complementary strengths and weaknesses across vision-language models at different visual scales, and we show that leveraging this modular structure achieves consistently higher accuracy than monolithic approaches. On the visually dense ScreenSpot-Pro benchmark, MEGA-GUI attains 73.18% accuracy, and on the semantically complex OSWorld-G benchmark it reaches 68.63%, surpassing previously reported results. Code and the Grounding Benchmark Toolkit (GBT) are available at https://github.com/samsungsds-research-papers/mega-gui.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。