arXiv:2510.09038cs.AIcs.CL2025-10被引 14

用连续嵌入记忆提升GUI智能体泛化能力,效果媲美闭源大模型。

Auto-scaling Continuous Memory for GUI Agent

  • 用视觉语言模型编码轨迹为固定长度连续向量,保留精细视觉信息。
  • 100k+轨迹训练下,长任务成功率显著提升,性能随记忆增长而持续优化。
  • 自扩展数据飞轮低成本构建数据集,仅微调1.2%参数即达顶尖效果。

我们研究如何为GUI智能体赋予可扩展的记忆机制,以提升其在陌生界面和长时序任务中的泛化能力。以往方法将历史轨迹压缩为文本标记,导致上下文过长且丢失关键视觉线索(如控件精确尺寸与位置)。本文提出一种连续记忆机制,利用视觉语言模型自身作为编码器,将每条GUI轨迹映射为固定长度的连续嵌入向量,并直接输入主干网络,显著降低上下文开销的同时保留细粒度视觉信息。随着记忆规模和检索深度增加,性能单调提升,优于依赖文本记忆的模型在长提示下的退化现象。为低成本扩展记忆,我们设计自动扩展数据飞轮:(i) 通过搜索发现新环境,(ii) 使用开源VLM合成任务,(iii) 由智能体执行轨迹,(iv) 用同一VLM验证成功。该流程共收集超10万条轨迹,成本约4000美元,仅对记忆编码器进行微调(在Q-Former上使用LoRA,参数占比1.2%),使用1500样本即可。在真实世界GUI基准测试中,该记忆增强型智能体在长周期任务与分布外场景下均表现出色。值得注意的是,Qwen-2.5-VL-7B + 连续记忆的表现可媲美最先进的闭源模型(如GPT-4o、Claude-4)。

原文摘要 · Abstract (English)

We study how to endow GUI agents with scalable memory that help generalize across unfamiliar interfaces and long-horizon tasks. Prior GUI agents compress past trajectories into text tokens, which balloons context length and misses decisive visual cues (e.g., exact widget size and position). We propose a continuous memory that encodes each GUI trajectory into a fixed-length sequence of continuous embeddings using the VLM itself as an encoder; these embeddings are plugged directly into the backbone's input layer, sharply reducing context cost while preserving fine-grained visual information. As memory size and retrieval depth increase, performance improves monotonically, unlike text memories that degrade with long prompts. To grow memory at low cost, we introduce an auto-scaling data flywheel that (i) discovers new environments via search, (ii) synthesizes tasks with an open-source VLM, (iii) rolls out trajectories with the agent, and (iv) verifies success with the same VLM. Using this pipeline, we collect 100k+ trajectories for about \$4000 and fine-tune only the memory encoder (LoRA on a Q-Former, 1.2\% parameters) with 1,500 samples. On real-world GUI benchmarks, our memory-augmented agent consistently improves success rates under long horizons and distribution shifts. Notably, Qwen-2.5-VL-7B + continuous memory achieves performance comparable to state-of-the-art closed-source models (e.g., GPT-4o, Claude-4).

GUI智能体连续记忆VLM自动化扩展

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。