用截图训练出顶尖的UI智能体,仅靠少量高质量数据实现高精度操作。
UI-Venus Technical Report: Building High-performance UI Agents with RFT
- 基于多模态大模型和强化微调,仅用几十万样本训练。
- 在屏幕定位与导航任务上超越现有最佳模型,72B版成功率超60%。
- 提出自进化轨迹对齐与稀疏动作增强,提升复杂任务规划能力。
我们提出UI-Venus,一个仅以截图为输入的原生UI智能体,基于Qwen2.5-VL通过强化微调(RFT)训练。该模型仅使用数十万高质量样本,在屏幕定位与导航任务上达到当前最优性能:7B和72B版本在Screenspot-V2/Pro基准上分别取得94.1%/50.8%和95.3%/61.9%准确率,超越开源GTA1与闭源UI-TARS-1.5。在AndroidWorld在线导航评测中,7B与72B版本成功率达49.1%与65.9%,同样领先。为此,我们设计了针对定位与导航的任务奖励函数及高效数据清洗策略。为进一步提升导航性能,提出自进化轨迹历史对齐与稀疏动作增强机制,优化历史推理路径并平衡关键动作分布,增强复杂任务中的连贯性与泛化能力。贡献包括开源首个达到顶尖水平的UI智能体、完整数据清洗流程与新型自进化框架,推动社区发展。代码已公开于https://github.com/inclusionAI/UI-Venus。
原文摘要 · Abstract (English)
We present UI-Venus, a native UI agent that takes only screenshots as input based on a multimodal large language model. UI-Venus achieves SOTA performance on both UI grounding and navigation tasks using only several hundred thousand high-quality training samples through reinforcement finetune (RFT) based on Qwen2.5-VL. Specifically, the 7B and 72B variants of UI-Venus obtain 94.1% / 50.8% and 95.3% / 61.9% on the standard grounding benchmarks, i.e., Screenspot-V2 / Pro, surpassing the previous SOTA baselines including open-source GTA1 and closed-source UI-TARS-1.5. To show UI-Venus's summary and planing ability, we also evaluate it on the AndroidWorld, an online UI navigation arena, on which our 7B and 72B variants achieve 49.1% and 65.9% success rate, also beating existing models. To achieve this, we introduce carefully designed reward functions for both UI grounding and navigation tasks and corresponding efficient data cleaning strategies. To further boost navigation performance, we propose Self-Evolving Trajectory History Alignment & Sparse Action Enhancement that refine historical reasoning traces and balances the distribution of sparse but critical actions, leading to more coherent planning and better generalization in complex UI tasks. Our contributions include the publish of SOTA open-source UI agents, comprehensive data cleaning protocols and a novel self-evolving framework for improving navigation performance, which encourage further research and development in the community. Code is available at https://github.com/inclusionAI/UI-Venus.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。