用结构化记忆减少冗余,让GUI代理更高效可靠。
MGA: Memory-Driven GUI Agent for Observation-Centric Interaction
- 通过观察-记忆双机制,分离任务决策与历史依赖。
- 在OSWorld上实现媲美复杂模型的性能,延迟更低。
- 适合追求轻量、稳定自动化系统的开发者。
多模态大语言模型虽推动了GUI代理发展,但长时程自动化仍受限于两大瓶颈:原始序列轨迹带来的上下文过载,以及过度设计专家模块导致的架构冗余。现有端到端与多智能体范式因拼接视觉-文本历史易引发错误传播,且冗余组件导致高推理延迟,限制实际应用。为此,我们提出内存驱动的GUI代理(MGA),一种极简框架,将长时程轨迹解耦为独立决策步骤,并由结构化状态记忆连接。MGA遵循‘先观察、再增强记忆’原则,依托两个紧密耦合的核心机制:(1) 观察器模块作为无任务、无意图的屏幕状态读取器,从根源上消除确认偏差、视觉幻觉和感知偏差;(2) 结构化记忆机制对每步交互进行提炼、验证与压缩,生成经验证的状态差值,构建轻量级状态转移链,避免无关历史干扰与系统冗余。通过以紧凑的事实性记忆过渡替代原始历史聚合,MGA显著降低认知负担与系统复杂度。在OSWorld及真实场景的大量实验表明,MGA在开放式GUI任务中表现优异,同时保持架构简洁,为下一代GUI自动化提供了可扩展、高效的蓝图。
原文摘要 · Abstract (English)
Multimodal Large Language Models (MLLMs) have significantly advanced GUI agents, yet long-horizon automation remains constrained by two critical bottlenecks: context overload from raw sequential trajectory dependence and architectural redundancy from over-engineered expert modules. Prevailing End-to-End and Multi-Agent paradigms struggle with error cascades caused by concatenated visual-textual histories and incur high inference latency due to redundant expert components, limiting their practical deployment. To address these issues, we propose the Memory-Driven GUI Agent (MGA), a minimalist framework that decouples long-horizon trajectories into independent decision steps linked by a structured state memory. MGA operates on an ``Observe First and Memory Enhancement`` principle, powered by two tightly coupled core mechanisms: (1) an Observer module that acts as a task-agnostic, intent-free screen state reader to eliminate confirmation bias, visual hallucinations, and perception bias at the root; and (2) a Structured Memory mechanism that distills, validates, and compresses each interaction step into verified state deltas, constructing a lightweight state transition chain to avoid irrelevant historical interference and system redundancy. By replacing raw historical aggregation with compact, fact-based memory transitions, MGA drastically reduces cognitive overhead and system complexity. Extensive experiments on OSWorld and real-world applications demonstrate that MGA achieves highly competitive performance in open-ended GUI tasks while maintaining architectural simplicity, offering a scalable and efficient blueprint for next-generation GUI automation {https://github.com/MintyCo0kie/MGA4OSWorld}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。