通过动态聚焦界面区域提升视觉模型的网页操作准确性
Visual Test-time Scaling for GUI Agent Grounding
- 动态缩放关键区域，减少视觉干扰
- 在Screenspot-pro和WebVoyager上分别提升28%和24%性能
- 适合需要高精度网页交互的智能代理应用
我们提出RegionFocus，一种面向视觉语言模型代理的视觉测试时扩展方法。由于GUI图像的视觉复杂性和大量界面元素，理解网页并准确选择操作具有挑战性。该方法通过动态放大相关区域，降低背景干扰，提升定位准确性。为此，我们设计了图像即地图机制，每一步可视化关键地标，提供透明的操作记录，并帮助代理有效筛选操作候选。即使采用简单的区域选择策略，在两个先进开源视觉语言模型代理UI-TARS和Qwen2.5-VL基础上，仍实现Screenspot-pro上超过28%、WebVoyager上超过24%的性能提升，凸显视觉测试时扩展在交互场景中的有效性。通过将RegionFocus应用于Qwen2.5-VL-72B模型，我们在ScreenSpot-Pro基准上达到61.6%的新最佳定位性能。代码将公开于https://github.com/tiangeluo/RegionFocus。
原文摘要 · Abstract (English)
We introduce RegionFocus, a visual test-time scaling approach for Vision Language Model Agents. Understanding webpages is challenging due to the visual complexity of GUI images and the large number of interface elements, making accurate action selection difficult. Our approach dynamically zooms in on relevant regions, reducing background clutter and improving grounding accuracy. To support this process, we propose an image-as-map mechanism that visualizes key landmarks at each step, providing a transparent action record and enables the agent to effectively choose among action candidates. Even with a simple region selection strategy, we observe significant performance gains of 28+\% on Screenspot-pro and 24+\% on WebVoyager benchmarks on top of two state-of-the-art open vision language model agents, UI-TARS and Qwen2.5-VL, highlighting the effectiveness of visual test-time scaling in interactive settings. We achieve a new state-of-the-art grounding performance of 61.6\% on the ScreenSpot-Pro benchmark by applying RegionFocus to a Qwen2.5-VL-72B model. Our code will be released publicly at https://github.com/tiangeluo/RegionFocus.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。