小模型也能在设备端高效操作图形界面，性能接近大模型。
Ferret-UI Lite: Lessons from Building Small On-Device GUI Agents
- 用真实与合成数据训练30亿参数小模型，支持跨平台操作。
- 在多个基准上实现91.6%的界面定位准确率和28%的导航成功率。
- 适合研究轻量化GUI智能体或开发本地化交互应用的开发者。
构建能有效与图形用户界面（GUI）交互的自主智能体仍是开放难题，尤其对小型设备端模型而言。本文提出Ferret-UI Lite，一个紧凑、端到端的GUI代理，可在移动、网页和桌面等多种平台上运行。通过优化小型模型的训练方法，我们基于真实与合成数据混合构建了3B参数的Ferret-UI Lite，利用思维链推理与视觉工具使用提升推理能力，并采用设计奖励的强化学习进行训练。该模型在多项任务中表现优异：在GUI定位任务中，于ScreenSpot-V2、ScreenSpot-Pro和OSWorld基准上分别达到91.6%、53.3%和61.2%的准确率；在导航任务中，安卓平台成功率达28.0%，OSWorld平台为19.8%。我们分享了构建轻量级、设备端运行的GUI智能体的方法与经验。
原文摘要 · Abstract (English)
Developing autonomous agents that effectively interact with Graphic User Interfaces (GUIs) remains a challenging open problem, especially for small on-device models. In this paper, we present Ferret-UI Lite, a compact, end-to-end GUI agent that operates across diverse platforms, including mobile, web, and desktop. Utilizing techniques optimized for developing small models, we build our 3B Ferret-UI Lite agent through curating a diverse GUI data mixture from real and synthetic sources, strengthening inference-time performance through chain-of-thought reasoning and visual tool-use, and reinforcement learning with designed rewards. Ferret-UI Lite achieves competitive performance with other small-scale GUI agents. In GUI grounding, Ferret-UI Lite attains scores of $91.6\%$, $53.3\%$, and $61.2\%$ on the ScreenSpot-V2, ScreenSpot-Pro, and OSWorld-G benchmarks, respectively. For GUI navigation, Ferret-UI Lite achieves success rates of $28.0\%$ on AndroidWorld and $19.8\%$ on OSWorld. We share our methods and lessons learned from developing compact, on-device GUI agents.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。