修复安卓界面代理评测基准缺陷,模型性能提升15%至75%
AndroidControl-Curated: Revealing the True Potential of GUI Agents through Benchmark Purification
- 清理原基准中的模糊与错误,构建更精准的评测体系
- 新基准下顶尖模型成功率升至75%,较之前提升15%
- 小模型仅30亿参数却媲美2350亿参数大模型
本地虚拟助手如Siri和Google Assistant日益重要,但受限于依赖开发者的固定API。GUI代理提供无API依赖的替代方案,但其应用因性能不佳而受阻——即使最佳模型(如Qwen3-VL-235B)在AndroidControl等基准上成功率也仅约60%,难以满足实际需求。本研究发现,问题不仅在于模型,更在于基准本身存在歧义与事实性错误,系统性低估了代理能力。为此,我们通过严谨净化流程改进AndroidControl,形成新版AndroidControl-Curated。在此新基准上,顶尖模型在复杂任务上的成功率达75%(提升15%),表明本地GUI代理实际已接近可用。我们推出新SOTA模型Magma-R1-3B,仅用2.4k条净化样本、60小时H20 GPU训练(约60美元),虽参数量仅为Qwen3-VL-235B的1/200,性能却相当。我们开源AndroidControl-Curated与Magma-R1模型,推动社区采用更真实评估体系,加速鲁棒本地助手发展。
原文摘要 · Abstract (English)
On-device virtual assistants like Siri and Google Assistant are increasingly pivotal, yet their capabilities are hamstrung by a reliance on rigid, developer-dependent APIs. GUI agents offer a powerful, API-independent alternative, but their adoption is hindered by the perception of poor performance, as even the best models (e.g. Qwen3-VL-235B) scores are capped at around 60% on benchmarks like AndroidControl, far from viability for real-world use. Our research reveals that issue lies not only with the models but with the benchmarks themselves. We identified notable shortcomings in AndroidControl, including ambiguities and factual errors, which systematically underrates agent capabilities. To address this critical oversight, we enhanced AndroidControl into AndroidControl-Curated, a refined version of the benchmark improved through a rigorous purification pipeline. On this enhanced benchmark, state-of-the-art models achieve success rates nearing 75% on complex tasks (15% improvement), reflecting that on-device GUI agents are actually closer to practical deployment than previously thought. We introduce our new SOTA model, Magma-R1- 3B, post-trained on just 2.4k curated samples using 60 hours of an H20 GPU (approximately $60). Despite being 200 times smaller in parameters, this model delivers performance comparable to Qwen3- VL-235B. We release both AndroidControl-Curated benchmark and Magma-R1 model to the research community, encouraging adoption of this enhanced benchmark to better reflect model capabilities and accelerate the development of robust, on-device virtual assistants.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。