构建测试基准与数据生成框架，提升GUI智能体错误恢复能力
Recovering Policy-Induced Errors: Benchmarking and Trajectory Synthesis for Robust GUI Agents
- 设计树状管道合成80万条含多样化错误的轨迹数据
- 32B模型在OSWorld上达47.4%成功率和33.8%全通过率
- 适合研究鲁棒性增强与自动化测试的开发者
尽管GUI智能体发展迅速，但往往缺乏自我纠错能力，限制了实际部署。为填补评估与数据层面的差距，我们提出GUI-RobustEval基准与基于驱动鲁棒性的轨迹合成方法（RoTS）。GUI-RobustEval包含1,216个可执行测试用例，系统评估多种真实场景下的错误恢复能力。在数据层面，RoTS通过树状管道主动发现多样错误模式并合成对应恢复步骤，生成80万条高质量数据。基于该数据微调的两个模型（RoTS-7B与RoTS-32B）在GUI-RobustEval及传统基准上均显著提升。尤其，RoTS-32B在OSWorld上达到47.4%成功率和33.8% All-Pass@4，表明更强的长序列错误恢复能力有助于提升整体性能。代码已开源。
原文摘要 · Abstract (English)
While GUI agents have advanced rapidly, they often lack the robustness to recover from their own errors, hindering real-world deployment. To bridge this gap at both the evaluation and data levels, we introduce GUI-RobustEval and propose Robustness-driven Trajectory Synthesis. GUI-RobustEval contains $1,216$ executable test cases that systematically measure error recovery capabilities across a broad and realistic spectrum of error modes. At the data level, RoTS is a scalable synthesis framework that creates $800k$ high-quality data via a tree-based pipeline that proactively discovers diverse error modes and synthesizes corresponding recovery steps. Our two models, RoTS-7B and RoTS-32B, fine-tuned on our dataset, both demonstrate significant gains on GUI-RobustEval and traditional GUI benchmarks. Notably, RoTS-32B achieves state-of-the-art performance on OSWorld, with a $47.4\%$ success rate and a $33.8\%$ All-Pass@4 score, suggesting that improved long-horizon error recovery ability contributes to both robustness and overall performance. Our code is available at https://github.com/AlibabaResearch/RoTS.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。