arXiv:2604.25380cs.CV2026-04被引 1

新基准+新模型,让界面智能体在动态环境中更靠谱

Benchmarking and Improving GUI Agents in High-Dynamic Environments

论文配图:Benchmarking and Improving GUI Agents in High-Dynamic Environments
图 1 · 摘自论文原文
  • 用视频输入+动态感知,捕捉界面变化中的关键信息
  • 在10个应用上测试,准确率显著优于传统方法
  • 适合做复杂动态界面交互的开发者和研究者

近年来,图形用户界面(GUI)智能体主要聚焦于监督微调(SFT)和强化学习(RL)训练范式。然而,高动态界面环境的挑战仍被严重低估。现有智能体通常仅依赖每次动作后的单张截图进行决策,导致状态不完全可观测,关键界面信息常被遗漏。为此,我们提出 DynamicGUIBench,一个涵盖十种应用、多种交互场景的在线GUI基准,其特征是在动作之间存在重要界面变化。同时,我们设计 DynamicUI 智能体,以交互过程的屏幕录制视频为输入,包含三个模块:动态感知器、精炼策略与反思模块。动态感知器对界面视频帧聚类,生成质心描述,并迭代选择最富含信息的帧作为动态上下文;针对所选帧与文本上下文之间的不一致与噪声,精炼策略采用动作条件过滤,减少思维-动作偏差与冗余;基于优化后的智能体轨迹,反思模块提供精准引导。在 DynamicGUIBench 上的实验表明,DynamicUI 显著提升动态界面环境下的性能,同时在其他公开基准上保持竞争力。

原文摘要 · Abstract (English)

Recent advancements in Graphical User Interface (GUI) agents have predominantly focused on training paradigms like supervised fine-tuning (SFT) and reinforcement learning (RL). However, the challenge of high-dynamic GUI environments remains largely underexplored. Existing agents typically rely on a single screenshot after each action for decision-making, leading to a partially observable (or even unobservable) Markov decision process, where the key GUI state including important information for actions is often inadequately captured. To systematically explore this challenge, we introduce DynamicGUIBench, a comprehensive online GUI benchmark spanning ten applications and diverse interaction scenarios characterized by important interface changes between actions. Furthermore, we present DynamicUI, an agent designed for dynamic interfaces, which takes screen-recording videos of the interaction process as input and consists of three components: a dynamic perceiver, a refinement strategy, and a reflection. Specifically, the dynamic perceiver clusters frames of the GUI video, generates captions for the centroids, and iteratively selects the most informative frames as the salient dynamic context. Considering that there may be inconsistencies and noise between the selected frames and the textual context of the agent, the refinement strategy employs an action-conditioned filtering to refine thoughts to mitigate thought-action inconsistency and redundancy. Based on the refined agent trajectories, the reflection module provides effective and accurate guidance for further actions. Experiments on DynamicGUIBench demonstrate that DynamicUI significantly improves the performance in dynamic GUI environments, while maintaining competitive performance on other public benchmarks.

GUI智能体动态界面视频输入多模态

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。