arXiv:2606.31410cs.AI2026-06被引 2

让AI在真实手机上自主操作,解决测试环境与实际使用脱节的问题。

Xiaomi-GUI-0 Technical Report

论文配图:Xiaomi-GUI-0 Technical Report
图 1 · 摘自论文原文
  • 用真机为主、沙盒为辅的混合架构,确保训练和测试环境贴近真实场景。
  • 在真实手机上实现72%的任务成功率,显著提升异常状态识别能力。
  • 通过失败轨迹自动生成修正数据,形成持续优化的闭环系统。

图形用户界面(GUI)智能体基于视觉语言模型,在真实应用中通过点击、滑动、输入等操作端到端完成用户任务。然而,现有方法主要在离线轨迹、模拟环境和标准化基准上训练评估,与真实应用在界面布局、交互逻辑及异常状态分布上差异显著,无法准确反映实际部署中的执行稳定性。账户状态、权限弹窗、支付验证和风控机制不断改变状态分布,导致基准得分与真实可用性之间存在持续差距。为弥合这一鸿沟,我们提出小米GUI-0,一种面向真实移动环境的原生多模态GUI智能体,其训练与评估均在真实设备闭环中进行。核心是真机主导的混合架构:物理设备作为主要执行环境,沙盒提供辅助支持,使数据收集、训练、推理与评估共享接近真实部署的执行分布。构建了涵盖高频头部任务、长尾意图的高泛化数据,以及用于反思与记忆的能力增强数据,并引入错误驱动的数据飞轮机制,将失败轨迹转化为修正动作、反思解释和恢复演示。模型通过监督微调、逐步强化学习与智能体强化学习三阶段渐进式训练。在公开基准与自研RealMobile数据集上评估,小米GUI-0在RealMobile上取得72.0%的成功率,在AndroidWorld上达78.9%,显著提升真实任务中的执行稳定性和异常状态识别能力。

原文摘要 · Abstract (English)

Graphical user interface (GUI) agents build on vision-language models to complete user tasks end-to-end in real applications through interface actions such as tapping, swiping, text entry, and navigation. However, existing GUI agents are trained and evaluated largely on offline trajectories, simulated environments, and standardized benchmarks. These differ substantially from real applications in interface layout, interaction logic, and abnormal-state distribution, and cannot faithfully characterize execution stability in real-world use, where account states, permission dialogs, payment authentication, and risk control continually reshape the state distribution and open a persistent gap between benchmark scores and real usability. To close this gap, we propose Xiaomi-GUI-0, a native multimodal GUI agent for real mobile environments, trained and evaluated within a real-device closed loop. At its core is a real-device-dominant hybrid infrastructure, where physical devices are the primary execution environment and sandboxes provide auxiliary support, so that data collection, training, rollout, and evaluation share an execution distribution close to real deployment. We construct multi-source training data spanning high-frequency head tasks, high-generalization data for long-tail intents, and capability-enhancement data for reflection and memory, and introduce an error-driven data flywheel that turns failure trajectories into corrected actions, reflective explanations, and recovery demonstrations. The model is trained through a progressive three-stage pipeline of supervised fine-tuning, step-level reinforcement learning, and agentic reinforcement learning. Evaluated on public benchmarks and our in-house RealMobile, Xiaomi-GUI-0 achieves 72.0% success on RealMobile and 78.9% on AndroidWorld, while substantially improving execution stability and abnormal-state recognition in real-world tasks.

GUI智能体真实设备多模态闭环训练

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。