用自适应四叉树压缩GUI截图冗余信息,提升AI助手效率
AQuaUI: Visual Token Reduction for GUI Agents with Adaptive Quadtrees

- 基于四叉树动态分割截图,保留关键区域代表令牌
- 在不重新训练的前提下实现13%加速与29%令牌减少
- 适合需要高效推理的GUI自动化场景
大型多模态模型(LMMs)已成为GUI代理模型的核心,但高分辨率截图在每一步迭代中引入大量冗余信息。这些截图的空间信息密度极不均匀:大片区域视觉同质且信息量低,而文本和图标需高保真度。现有方法或需额外训练,或依赖注意力压缩,忽视了GUI截图的结构布局与空间冗余。本文提出AQuaUI,一种无需训练的推理阶段令牌压缩方法,利用截图的信息密度非均匀性。AQuaUI在每个截图上构建自适应四叉树,每个叶节点保留一个合并后的代表令牌,并保持保留令牌的空间位置一致性。为进一步提升多步交互中的时序一致性,提出条件四叉树算法,利用连续截图间的连续性,以先前四叉树为参考优化当前结构,从而在静态或轻微移动的GUI状态下保留细粒度区域。在主流基准测试中,AQuaUI在无需重训练的情况下显著提升精度-效率权衡。例如,在GUI-Owl-1.5-32B-Instruct上,实现最高13.22%加速、29.52%更少视觉令牌,同时保持99.06%的完整令牌性能,表明可直接利用GUI截图的空间冗余进行推理优化。
原文摘要 · Abstract (English)
Large Multimodal Models (LMMs) have recently emerged as promising backbones for GUI-agent models, where high-resolution GUI screenshots are introduced to the prompts at each iteration step. However, these screenshots exhibit highly non-uniform spatial information density: large regions may carry little information and are visually homogeneous, while key text and icons may require high visual fidelity. Existing approaches to this problem either require additional training or rely on attention-based token compression, ignoring the structured layout and spatial redundancy of GUI screenshots. To fill the gap, this paper proposes AquaUI, a training-free inference-time token reduction method for GUI agent models that utilizes the non-uniform information density in screenshots. AQuaUI constructs an adaptive quadtree on each screenshot input and keeps one representative merged token per leaf of the quadtree. AQuaUI preserves the spatial positions of retained tokens throughout the pipeline to ensure that all position-encoding stages remain consistent. To further improve temporal consistency across multi-step GUI interactions, we propose a conditional quadtree algorithm that leverages the continuity between consecutive screenshots within a single request. Specifically, it refines the current quadtree using previous quadtrees as references, helping preserve fine-grained regions across static or mildly shifted GUI states. We implement AQuaUI on state-of-the-art GUI agent models and conduct experiments on standard grounding and navigational benchmarks. AQuaUI consistently shows improved accuracy-efficiency trade-offs over prior baselines. Notably, on GUI-Owl-1.5-32B-Instruct, AQuaUI achieves up to 13.22% speedup and 29.52% fewer visual tokens while retaining 99.06% of full-token performance, suggesting that the spatial redundancy of GUI screenshots can be exploited at inference without retraining.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。