通过空间时间感知的缓存压缩,让图形界面智能体更快更省资源。
GUI-KV: Efficient GUI Agents via KV Cache with Spatio-Temporal Awareness
- 用视觉重要性+历史冗余度动态筛选缓存,不需重新训练。
- 在5张截图场景下,计算量降38.9%,任务准确率反升4.1%。
- 适合做自动化操作的AI助手开发,尤其关注效率与精度平衡者。
基于视觉语言模型的图形用户界面(GUI)智能体已成为自动化人机交互流程的有前景方案。然而,其处理高分辨率截图序列和长时序任务时面临效率瓶颈,推理慢、成本高且受内存限制。虽然键值(KV)缓存可缓解此问题,但图像密集场景下存储完整缓存不可行。现有缓存压缩方法因未考虑GUI的空间与时间冗余而效果不佳。本文分析了GUI智能体工作负载中的注意力模式,发现相较于自然图像,所有变压器层的注意力稀疏性均很高。这一发现启发我们采用统一预算分配策略,实证表明其优于复杂分层方案。在此基础上,提出GUI-KV——一种无需重训练的即插即用式缓存压缩方法,结合两项新技术:(i) 空间显著性引导,通过隐藏状态的L2范数增强注意力得分,更好保留语义重要视觉标记;(ii) 时间冗余评分,将先前帧的键投影到当前帧的键子空间,优先剔除冗余历史。在标准GUI智能体基准测试中,GUI-KV超越竞争性压缩基线,以小预算接近全缓存精度。值得注意的是,在AgentNetBench基准的5截图设置下,相比全缓存基线,解码浮点运算量减少38.9%,同时步骤准确率提升4.1%。结果表明,利用GUI特异性冗余可实现高效可靠的智能体性能。
原文摘要 · Abstract (English)
Graphical user interface (GUI) agents built on vision-language models have emerged as a promising approach to automate human-computer workflows. However, they also face the inefficiency challenge as they process long sequences of high-resolution screenshots and solving long-horizon tasks, making inference slow, costly and memory-bound. While key-value (KV) caching can mitigate this, storing the full cache is prohibitive for image-heavy contexts. Existing cache-compression methods are sub-optimal as they do not account for the spatial and temporal redundancy of GUIs. In this work, we first analyze attention patterns in GUI agent workloads and find that, unlike in natural images, attention sparsity is uniformly high across all transformer layers. This insight motivates a simple uniform budget allocation strategy, which we show empirically outperforms more complex layer-varying schemes. Building on this, we introduce GUI-KV, a plug-and-play KV cache compression method for GUI agents that requires no retraining. GUI-KV combines two novel techniques: (i) spatial saliency guidance, which augments attention scores with the L2 norm of hidden states to better preserve semantically important visual tokens, and (ii) temporal redundancy scoring, which projects previous frames' keys onto the current frame's key subspace to preferentially prune redundant history. Across standard GUI agent benchmarks and models, GUI-KV outperforms competitive KV compression baselines, closely matching full-cache accuracy at modest budgets. Notably, in a 5-screenshot setting on the AgentNetBench benchmark, GUI-KV reduces decoding FLOPs by 38.9% while increasing step accuracy by 4.1% over the full-cache baseline. These results demonstrate that exploiting GUI-specific redundancies enables efficient and reliable agent performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。