用流模型让电脑界面操作更像人手,支持连续拖拽。
ShowUI-$π$: Flow-based Generative Models as GUI Dexterous Hands
- 统一离散点击与连续拖拽的生成模型,支持多模式交互。
- 通过轻量级专家模型实现平滑拖拽轨迹,提升操作稳定性。
- 构建2万条拖拽数据集和新评测基准,适合研究界面智能体。
构建具备灵巧操作能力的智能体,是实现在机器人与数字环境中类人自动化的重要一步。然而,现有图形界面(GUI)智能体依赖离散点击预测(x,y),无法实现自由形式的闭环轨迹(如拖动进度条),这类操作需要持续感知与实时调整。本文提出ShowUI-π,首个基于流模型的GUI灵巧操作代理,其设计包括:(i) 统一离散点击与连续拖拽动作,在同一模型中融合多种交互模式,实现灵活适应;(ii) 基于流的动作生成机制,通过轻量级动作专家从连续视觉观测中预测微小光标调整,确保轨迹平滑稳定;(iii) 构建包含2万条拖拽轨迹的训练数据集与新评测基准ScreenDrag,涵盖五个领域(如PowerPoint、Adobe Premiere Pro),提供在线与离线综合评估协议。实验表明,现有专有系统在ScreenDrag上仍表现不佳(如Operator得分为13.27,最佳Gemini-2.5-CUA为22.18),而ShowUI-π仅用450M参数即达26.98分,充分验证了方法的有效性与任务难度。代码已开源。
原文摘要 · Abstract (English)
Building intelligent agents capable of dexterous manipulation is essential for achieving human-like automation in both robotics and digital environments. However, existing GUI agents rely on discrete click predictions (x,y), which prohibits free-form, closed-loop trajectories (e.g. dragging a progress bar) that require continuous, on-the-fly perception and adjustment. In this work, we develop ShowUI-$π$, the first flow-based generative model as GUI dexterous hand, featuring the following designs: (i) Unified Discrete-Continuous Actions, integrating discrete clicks and continuous drags within a shared model, enabling flexible adaptation across diverse interaction modes; (ii) Flow-based Action Generation for drag modeling, which predicts incremental cursor adjustments from continuous visual observations via a lightweight action expert, ensuring smooth and stable trajectories; (iii) Drag Training data and Benchmark, where we manually collect and synthesize 20K drag trajectories across five domains (e.g. PowerPoint, Adobe Premiere Pro), and introduce ScreenDrag, a benchmark with comprehensive online and offline evaluation protocols for assessing GUI agents' drag capabilities. Our experiments show that proprietary GUI agents still struggle on ScreenDrag (e.g. Operator scores 13.27, and the best Gemini-2.5-CUA reaches 22.18). In contrast, ShowUI-$π$ achieves 26.98 with only 450M parameters, underscoring both the difficulty of the task and the effectiveness of our approach. We hope this work advances GUI agents toward human-like dexterous control in digital world. The code is available at https://github.com/showlab/showui-pi.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。