arXiv:2509.15566cs.CVcs.AI2025-09NeurIPS被引 5

模仿人类看界面的思考过程,让AI操作界面更自然。

BTL-UI: Blink-Think-Link Reasoning Model for GUI Agent

  • 分三步:快速扫视、思考决策、生成操作指令。
  • 在静态和动态任务中表现优异,超越现有方法。
  • 适合做智能自动化工具或人机交互研究者参考。

在人工智能驱动的人机图形界面交互自动化领域,尽管多模态大模型和强化微调技术取得了显著进展,但其交互逻辑仍与人类自然使用界面的方式存在显著差异。为此,我们提出“眨眼-思考-链接”(Blink-Think-Link, BTL)框架,模拟人类在与图形界面交互时的认知过程。该框架将交互分解为三个生物上合理的阶段:(1) Blink——快速检测并聚焦屏幕相关区域,类比眼球快速扫视;(2) Think——高层次推理与决策,对应认知规划;(3) Link——生成可执行命令以实现精准动作控制,模拟人类动作选择机制。此外,我们引入两项关键技术:(1) Blink数据生成——专为眨眼阶段设计的自动化标注流水线;(2) BTL奖励——首个基于规则的奖励机制,支持过程与结果双重驱动的强化学习。基于此框架,我们构建了GUI代理模型BTL-UI,其在多个综合基准测试中,在静态界面理解与动态交互任务上均展现出竞争力。这些结果为该框架在构建先进GUI代理中的有效性提供了充分的实证支持。

原文摘要 · Abstract (English)

In the field of AI-driven human-GUI interaction automation, while rapid advances in multimodal large language models and reinforcement fine-tuning techniques have yielded remarkable progress, a fundamental challenge persists: their interaction logic significantly deviates from natural human-GUI communication patterns. To fill this gap, we propose "Blink-Think-Link" (BTL), a brain-inspired framework for human-GUI interaction that mimics the human cognitive process between users and graphical interfaces. The system decomposes interactions into three biologically plausible phases: (1) Blink - rapid detection and attention to relevant screen areas, analogous to saccadic eye movements; (2) Think - higher-level reasoning and decision-making, mirroring cognitive planning; and (3) Link - generation of executable commands for precise motor control, emulating human action selection mechanisms. Additionally, we introduce two key technical innovations for the BTL framework: (1) Blink Data Generation - an automated annotation pipeline specifically optimized for blink data, and (2) BTL Reward -- the first rule-based reward mechanism that enables reinforcement learning driven by both process and outcome. Building upon this framework, we develop a GUI agent model named BTL-UI, which demonstrates competitive performance across both static GUI understanding and dynamic interaction tasks in comprehensive benchmarks. These results provide conclusive empirical validation of the framework's efficacy in developing advanced GUI Agents.

GUI代理认知模型强化学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。