让界面智能体学会回顾历史操作,提升长任务中的决策能力。
History-Aware Reasoning for GUI Agents
- 通过反思错误和设计修正策略,增强智能体的短期记忆。
- 在多基准测试中显著提升长周期界面任务的执行成功率。
- 适合需要连续操作的自动化场景,如跨应用流程处理。
多模态大模型的发展极大推动了图形用户界面(GUI)自动化。为弥合用户简短指令与真实执行复杂性之间的差距,赋予GUI智能体可靠的回忆推理能力至关重要。现有方法结合强化学习与系统2思维链,在推理能力上取得显著进展。然而,对于长周期GUI任务,历史交互将每个屏幕与目标导向的执行链关联起来,有效利用这些线索对当前决策至关重要。现有原生GUI智能体在显式推理中表现出弱短期记忆,将串联交互视为离散屏幕理解,即忽视了任务过程中的历史信息。这种无视历史的推理方式制约了其在GUI自动化中的表现。为此,我们提出历史感知推理(HAR)框架,通过定制化策略促使智能体反思自身错误,并从中获取情景推理知识,从而增强长周期交互中的短期记忆。该框架主要包括构建反思学习场景、合成定制化修正指南以及设计混合强化学习奖励函数。基于HAR框架,我们开发了一个原生端到端模型HAR-GUI-3B,将智能体的内在推理模式从无视历史转变为关注历史,赋予其稳定的短期记忆和对屏幕细节的可靠感知。在多个GUI相关基准上的全面评估表明,该方法具有高效性和泛化能力。
原文摘要 · Abstract (English)
Advances in Multimodal Large Language Models have significantly enhanced Graphical User Interface (GUI) automation. Equipping GUI agents with reliable episodic reasoning capabilities is essential for bridging the gap between users' concise task descriptions and the complexities of real-world execution. Current methods integrate Reinforcement Learning (RL) with System-2 Chain-of-Thought, yielding notable gains in reasoning enhancement. For long-horizon GUI tasks, historical interactions connect each screen to the goal-oriented episode chain, and effectively leveraging these clues is crucial for the current decision. However, existing native GUI agents exhibit weak short-term memory in their explicit reasoning, interpreting the chained interactions as discrete screen understanding, i.e., unawareness of the historical interactions within the episode. This history-agnostic reasoning challenges their performance in GUI automation. To alleviate this weakness, we propose a History-Aware Reasoning (HAR) framework, which encourages an agent to reflect on its own errors and acquire episodic reasoning knowledge from them via tailored strategies that enhance short-term memory in long-horizon interaction. The framework mainly comprises constructing a reflective learning scenario, synthesizing tailored correction guidelines, and designing a hybrid RL reward function. Using the HAR framework, we develop a native end-to-end model, HAR-GUI-3B, which alters the inherent reasoning mode from history-agnostic to history-aware, equipping the GUI agent with stable short-term memory and reliable perception of screen details. Comprehensive evaluations across a range of GUI-related benchmarks demonstrate the effectiveness and generalization of our method.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。