让GUI自动化任务计划可见可改，提升可控性与透明度。
Plover: Steering GUI Agents through Plan-Centric Interaction
- 将任务计划外化为可查看、可编辑的持久化文档
- 支持局部修正与自然语言指导，修复失败案例成功率显著提升
- 适合需要监督和干预自动化流程的研究者与开发者
图形用户界面（GUI）自动化在真实环境中仍具挑战性，动态布局、意外弹窗和界面状态变化常导致自主代理偏离用户意图。现有基于视觉的多模态代理虽通过截图与自然语言指令提升灵活性，但规划与适应过程多为内部隐式，限制了用户对系统行为的检查、监督与纠正。我们提出Plover，一种以计划为中心的视觉化GUI自动化系统，将任务计划与重规划作为持久、可检查、可修改的实体。通过规划器-执行器架构，Plover支持显式监督执行过程演变，通过可编辑计划实现局部修正，结合自然语言指导与基于截图的干预，同时保留修复前的进度。六名参与者参与的初步研究指导了交互设计。后续在基准故障修复与场景化工作流分析中评估Plover，结果表明：当计划可见且干预局部化时，多数自动代理失败可结构性修复；显式重规划显著提升了GUI自动化的透明性、可控性与适应性。
原文摘要 · Abstract (English)
Graphical user interface (GUI) automation remains challenging in real-world environments, where dynamic layouts, unexpected dialogs, and evolving interface states can cause autonomous agents to drift from user intent. Recent vision-based multimodal agents improve flexibility by operating directly over screenshots and natural language instructions, but planning and adaptation often remain internal, limiting users' ability to inspect, supervise, or correct system behavior. We present Plover, a plan-centric vision-based GUI automation system that externalizes task plans and replanning as persistent, inspectable, and revisable artifacts. Through a planner--executor architecture, Plover supports explicit supervision of evolving execution, localized correction through editable plans, natural-language guidance, and screenshot-grounded interventions, while preserving prior progress during repair. A formative study with six participants informed the interaction design. We then evaluate Plover through benchmark failure-case repair and scenario-based workflow analyses. Our results show that many autonomous GUI-agent failures are structurally repairable when plans remain visible and interventions are localized, and that explicit replanning helps make GUI automation more transparent, controllable, and adaptable.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。