构建可验证的长时序GUI任务基准，提升真实场景下AI助手的执行准确率。
NaturalGAIA: A Verifiable Benchmark and Hierarchical Framework for Long-Horizon GUI Tasks
- 分离逻辑路径与语言描述，模拟人类非线性认知和上下文依赖的真实操作意图。
- 提出45.6%加权路径成功率，显著优于基线21.1%，且减少75%令牌消耗和76%执行时间。
- 适合研究复杂交互任务的智能体系统，尤其关注效率与真实场景适配的开发者。
尽管大模型驱动的GUI代理取得显著进展，但高保真度真实感与可验证评估准确性之间仍存在矛盾。为此，我们提出NaturalGAIA，一个基于真实人类GUI交互意图的可验证评估数据集。通过将逻辑因果路径与语言叙事解耦，严格模拟自然的人类意图，其特征为认知非线性和上下文依赖性。此外，我们提出LightManus-Jarvis层级协作框架：LightManus负责动态拓扑规划与上下文演化，Jarvis则通过混合视觉-结构感知确保执行精度。实验表明，该方法在加权路径成功率上达到45.6%，显著优于当前最优基线（21.1%），同时降低75%的令牌消耗与76%的执行时间。结果验证了宏观规划与微观执行范式在处理复杂自然化任务中的有效性。代码已开源：https://github.com/KeLes-Coding/NatureGAIA。
原文摘要 · Abstract (English)
Despite significant advances in LLM-driven GUI agents, the field remains constrained by the challenge of reconciling high-fidelity realism with verifiable evaluation accuracy. To address this, we introduce NaturalGAIA, a verifiable evaluation dataset grounded in real-world human GUI interaction intents. By decoupling logical causal pathways from linguistic narratives, it rigorously simulates natural human intent, characterized by cognitive non-linearity and contextual dependencies. Furthermore, we propose LightManus-Jarvis, a hierarchical collaborative framework where LightManus manages dynamic topological planning and context evolution, while Jarvis~ensures execution precision via hybrid visual-structural perception. Experiments demonstrate that our approach achieves a Weighted Pathway Success Rate of 45.6%, significantly outperforming the state-of-the-art baseline (21.1%), while reducing token consumption by 75% and execution time by 76%. These results validate the efficacy of the macro-planning and micro-execution paradigm in handling complex naturalized tasks. Our code is publicly available at: https://github.com/KeLes-Coding/NatureGAIA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。