arXiv:2605.16024cs.AI2026-05

让桌面助手更懂模糊界面,自动识别该探索还是该行动

ScreenSearch: Uncertainty-Aware OS Exploration

论文配图:ScreenSearch: Uncertainty-Aware OS Exploration
图 1 · 摘自论文原文
  • 用结构化检索+去重图,构建跨应用的屏幕状态库
  • 发现模糊状态需主动探查,仅减少模糊不等于有效探索
  • 适合做桌面自动化、GUI智能体的开发者和研究者

桌面GUI代理在部分可观测环境下运行:视觉相似的界面可能对应不同工作流状态,局部合理操作可能导致截然不同的结果。本文将此问题建模为操作系统状态探索任务,有效行为需同时拓展可到达边界并降低不确定性。提出ScreenSearch系统,结合结构化屏幕检索与去重机制,以及基于不确定性的PUCT图强化学习算法,实现大规模桌面探索。检索层将UIA树转为带位置信息的结构特征,通过稀疏令牌搜索与元数据过滤索引相关屏幕,并在虚拟机工作节点间维护共享去重状态图。在此基础上,定义可扩展的模糊性信号:若相似屏幕在相同操作下产生不同下一状态,则该状态应进一步探测而非视为已解决。该信号与前沿奖励共同驱动大规模探索及重播起点策略评估。在11个桌面应用中,收集超过100万张截图和超3万条去重状态,形成具有显著跨应用与应用内多样性的探索语料库。固定重播起点实验显示明显的新颖性-模糊性权衡:某些策略快速降低模糊性但发现新状态少。单独追求模糊性降低不足以作为探索目标。附录消融实验表明,更强的提案先验能显著提升语料构建阶段的独特状态发现率。结果表明,状态识别、提案质量与模糊性感知搜索共同决定何时探测、何时执行。

原文摘要 · Abstract (English)

Desktop GUI agents operate under partial observability: visually similar screens can correspond to different underlying workflow states, so locally plausible actions can lead to sharply different outcomes. We frame this as a problem of computer/OS state exploration, where effective behavior requires both expanding the reachable frontier and reducing ambiguity before committing. We present ScreenSearch, a system that combines structural screen retrieval and deduplication with an ambiguity-aware PUCT graph-bandit for large-scale desktop exploration. The retrieval layer converts UIA trees into location-aware structural features, indexes related screens through sparse token search and metadata filters, and maintains a shared deduplicated state graph across VM workers. On top of this graph, we define a scalable ambiguity signal based on matched-action outcome dispersion. If similar screens produce different next states under the same action signature, the state should be probed further rather than treated as resolved. We use this signal together with frontier rewards to drive large-scale exploration and replay-start policy evaluation over the shared graph. Across 11 desktop applications, ScreenSearch collects over 1M screenshots and over 30K deduplicated states, yielding large exploration corpora with substantial cross-application and within-application diversity. On a fixed replay-start slice, we observe a clear novelty--ambiguity trade-off: some policies reduce ambiguity quickly while discovering little frontier. Ambiguity reduction alone is therefore not a sufficient exploration objective. Appendix ablations show that stronger proposal priors can materially improve unique-state discovery during corpus building. These results suggest that state identity, proposal quality, and ambiguity-aware search all matter when deciding when to probe and when to commit.

GUI智能体状态探索模糊性感知桌面自动化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。