RecAgent通过主动推荐组件和人机协同,降低界面操作中的模糊性。
Uncertainty-Aware GUI Agent: Adaptive Perception through Component Recommendation and Human-in-the-Loop Refinement
- 基于组件推荐聚焦关键界面元素,减少冗余输入干扰
- 在任务模糊时主动请求用户反馈,提升决策准确性
- 适合需要高可靠性的自动化操作场景,如手机智能助手
图形用户界面(GUI)代理在自动化移动端任务方面展现出潜力,但仍面临输入冗余和决策模糊的问题。本文提出RecAgent,一种不确定性感知的代理,通过自适应感知解决上述问题。我们区分两种不确定性:(1) 感知不确定性,由屏幕信息过载和噪声引起;(2) 决策不确定性,源于任务模糊和复杂推理。为降低感知不确定性,RecAgent采用组件推荐机制,识别并聚焦最相关UI元素;针对决策不确定性,引入交互模块,在模糊情境下请求用户反馈,实现意图感知决策。二者集成于统一框架中,主动降低输入复杂度,并在高不确定性情况下通过人机协同修正。此外,我们构建了名为ComplexAction的数据集,用于评估代理在复杂场景中执行指定单步动作的成功率。大量实验验证了方法的有效性。代码与数据集将公开于https://github.com/Fanye12/RecAgent。
原文摘要 · Abstract (English)
Graphical user interface (GUI) agents have shown promise in automating mobile tasks but still struggle with input redundancy and decision ambiguity. In this paper, we present \textbf{RecAgent}, an uncertainty-aware agent that addresses these issues through adaptive perception. We distinguish two types of uncertainty in GUI navigation: (1) perceptual uncertainty, caused by input redundancy and noise from comprehensive screen information, and (2) decision uncertainty, arising from ambiguous tasks and complex reasoning. To reduce perceptual uncertainty, RecAgent employs a component recommendation mechanism that identifies and focuses on the most relevant UI elements. For decision uncertainty, it uses an interactive module to request user feedback in ambiguous situations, enabling intent-aware decisions. These components are integrated into a unified framework that proactively reduces input complexity and reacts to high-uncertainty cases via human-in-the-loop refinement. Additionally, we propose a dataset called \textbf{ComplexAction} to evaluate the success rate of GUI agents in executing specified single-step actions within complex scenarios. Extensive experiments validate the effectiveness of our approach. The dataset and code will be available at https://github.com/Fanye12/RecAgent.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。