arXiv:2605.18048cs.AI2026-05被引 1

让AI助手主动查文档完成复杂操作,提升处理长尾任务能力

DocOS: Towards Proactive Document-Guided Actions in GUI Agents

论文配图:DocOS: Towards Proactive Document-Guided Actions in GUI Agents
图 1 · 摘自论文原文
  • AI助手主动搜索网页文档获取操作指引
  • 在真实浏览器环境中实现文档理解与操作执行
  • 适合需要自主学习新任务的智能交互系统

尽管图形用户界面(GUI)代理在自动化设备交互中表现良好,但主要依赖预训练或指令微调获得的静态参数化知识。这种依赖从根本上限制了其处理需要显式流程知识的长尾任务的能力,常迫使代理采用低效且脆弱的试错探索。为缓解这一局限,我们提出面向动态开放网络环境的「主动文档引导操作」新范式,模拟人类解决问题方式,使代理能自主搜索相关文档以解决长尾任务。为此,我们构建了DocOS基准,用于评估代理在完全交互环境中的文档引导问题求解能力。DocOS要求代理自主导航浏览器、定位相关在线文档、理解操作说明,并准确将其转化为可执行的GUI动作。大量实验表明,进展受限于双重瓶颈:代理在主动搜索中难以可靠定位相关信息,且常无法准确将获取的指令转化为精确动作,表明文档引导交互是实现动态环境中自演化GUI代理的关键路径。

原文摘要 · Abstract (English)

While Graphical User Interface (GUI) agents have shown promising performance in automated device interaction, they primarily depend on static parametric knowledge from pre-training or instruction tuning. This reliance fundamentally limits their ability to handle long-tailed tasks that require explicit procedural knowledge absent from model parameters, often forcing agents to resort to inefficient and brittle trial-and-error exploration. To mitigate this limitation, we introduce \textbf{Proactive Document-Guided Action} for GUI agents in dynamic, open-web environments, a novel paradigm that mirrors human problem-solving by enabling agents to autonomously search for relevant documentation to resolve long-tailed tasks. To evaluate agents' capability in this paradigm, we propose \textbf{DocOS}, a benchmark designed to assess document-guided problem solving in fully interactive environments. DocOS requires agents to autonomously navigate a web browser, locate relevant online documentation, comprehend procedural instructions, and faithfully ground them into executable GUI actions. Extensive experiments reveal that progress is strictly constrained by dual bottlenecks: agents struggle to reliably locate relevant information during proactive search and frequently fail to faithfully ground retrieved instructions into precise actions, pointing toward document-guided interaction as a crucial pathway for enabling self-evolving GUI agents in dynamic environments.

GUI代理文档引导长尾任务自主学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。