arXiv:2603.26041cs.CV2026-03

通过缓存截图特征,显著降低GUI导航的计算开销。

Where and How to Prune: An Empirical Study of Visual Token Pruning for GUI Agent Navigation

  • 缓存ViT编码避免重复计算,减少冗余计算量。
  • 保留前景与背景语义平衡,维持空间感知一致性。
  • 适合需要高效推理的GUI自动化系统开发者。

近年来,GUI代理在导航任务中展现出强大潜力,但完整保存历史截图会带来巨大计算开销。本文研究了将标记剪枝这一即插即用的推理加速技术应用于GUI代理导航场景。首先,我们探讨剪枝应发生在何处:发现现有方法忽略了一个系统级冗余——相同截图在不同步骤中被反复输入模型,导致其ViT编码被重复计算。我们证明,其ViT编码嵌入可完全缓存并跨步骤复用,大幅降低FLOPs且不损失模型性能。这一发现表明,推理加速应聚焦于后续的大语言模型(LLM)。在此基础上,进一步探讨如何在LLM内进行剪枝,提炼出两个关键见解:(i) 从语义角度看,标记预算应在前景与背景区域间保持平衡;(ii) 从空间角度看,保留的标记应保持空间均匀性,以维护模型的全局空间感知能力。这些发现为GUI代理导航中的推理加速与标记剪枝设计提供了实用指导。

原文摘要 · Abstract (English)

In recent years, GUI agents have demonstrated strong potential in navigation tasks. However, preserving complete historical screenshots introduces substantial computational overhead. This paper investigates how token pruning, a plug-and-play inference acceleration technique, can be effectively applied to GUI agent navigation scenarios. Firstly, we address the question of where pruning should occur. We identify a system-level redundancy overlooked by existing methods: as the same screenshot is repeatedly fed into the model across different steps, its ViT encoding is redundantly recomputed each time. We show that its ViT-encoded embeddings can be fully cached and reused across steps, substantially reducing FLOPs while preserving model performance. This finding suggests that inference acceleration efforts should focus on the subsequent Large Language Model (LLM). Building on this, we further address the question of how to prune within the LLM, and distill two key insights: (i) from a semantic perspective, the token budget should be balanced between foreground and background regions; (ii) from a spatial perspective, the spatial uniformity of retained tokens should be maintained to preserve the model's global spatial perception. These findings provide practical guidance for the design of inference acceleration and token pruning for GUI agent navigation.

GUI导航标记剪枝推理加速

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。