arXiv:2504.14239cs.AIcs.CL2025-04被引 138

让GUI代理从被动执行转向主动思考,提升复杂任务的规划与纠错能力。

InfiGUI-R1: Advancing Multimodal GUI Agents from Reactive Actors to Deliberative Reasoners

论文配图:InfiGUI-R1: Advancing Multimodal GUI Agents from Reactive Actors to Deliberative Reasoners
图 1 · 摘自论文原文
  • 通过两阶段训练,将代理从被动响应者升级为有意识推理者。
  • 在GUI任务中达到92.3%的轨迹准确率,显著优于现有方法。
  • 适合研究智能自动化、人机交互和大模型推理的开发者与学者。

多模态大语言模型(MLLM)已推动图形用户界面(GUI)代理的发展,在自动化计算设备任务方面展现出潜力。近期工作开始探索GUI任务中的推理,取得初步成效。然而,许多现有方法依赖人工设计的推理模板,导致推理在复杂GUI环境中缺乏鲁棒性和适应性。同时,部分代理仍作为被动响应者,依赖隐式推理,难以应对需要规划与错误恢复的任务。本文主张,推进代理发展需从被动响应转向主动推理。为此,我们提出InfiGUI-R1,一个基于MLLM的GUI代理,采用我们提出的Actor2Reasoner框架——一种以推理为核心的两阶段训练方法,逐步实现从被动响应者到主动推理者的转变。第一阶段‘推理注入’旨在建立基础推理能力,利用空间推理蒸馏技术,通过带有显式推理步骤的轨迹,将教师模型的跨模态空间推理能力迁移到MLLM中,使模型能在生成动作前融合视觉-空间信息与逻辑推理。第二阶段‘深思强化’通过强化学习将基础推理器优化为深度推理器,引入两种策略:子目标引导,奖励模型生成准确的中间子目标;错误恢复场景构建,从易错步骤中生成失败与恢复的训练场景。实验结果表明,InfiGUI-R1在GUI定位与轨迹任务中表现优异,轨迹准确率达92.3%。相关资源见https://github.com/Reallm-Labs/InfiGUI-R1。

原文摘要 · Abstract (English)

Multimodal Large Language Models (MLLMs) have powered Graphical User Interface (GUI) Agents, showing promise in automating tasks on computing devices. Recent works have begun exploring reasoning in GUI tasks with encouraging results. However, many current approaches rely on manually designed reasoning templates, which may result in reasoning that is not sufficiently robust and adaptive for complex GUI environments. Meanwhile, some existing agents continue to operate as Reactive Actors, relying primarily on implicit reasoning that may lack sufficient depth for GUI tasks demanding planning and error recovery. We argue that advancing these agents requires a shift from reactive acting towards acting based on deliberate reasoning. To facilitate this transformation, we introduce InfiGUI-R1, an MLLM-based GUI agent developed through our Actor2Reasoner framework, a reasoning-centric, two-stage training approach designed to progressively evolve agents from Reactive Actors to Deliberative Reasoners. The first stage, Reasoning Injection, focuses on establishing a basic reasoner. We employ Spatial Reasoning Distillation to transfer cross-modal spatial reasoning capabilities from teacher models to MLLMs through trajectories with explicit reasoning steps, enabling models to integrate GUI visual-spatial information with logical reasoning before action generation. The second stage, Deliberation Enhancement, refines the basic reasoner into a deliberative one using Reinforcement Learning. This stage introduces two approaches: Sub-goal Guidance, which rewards models for generating accurate intermediate sub-goals, and Error Recovery Scenario Construction, which creates failure-and-recovery training scenarios from identified prone-to-error steps. Experimental results show InfiGUI-R1 achieves strong performance in GUI grounding and trajectory tasks. Resources at https://github.com/Reallm-Labs/InfiGUI-R1.

GUI代理主动推理强化学习多模态模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。