构建图结构基准测试框架，实现复杂长时序移动任务的多路径评估。
ColorBench: Benchmarking Mobile Agents with Graph-Structured Framework for Complex Long-Horizon Tasks
- 用图结构建模真实设备交互状态，静态模拟动态行为。
- 包含175个任务，平均超过13步，支持多正确路径与错误路径分析。
- 适合评估移动智能体在复杂长任务中的鲁棒性与泛化能力。
多模态大语言模型的发展使智能体能够直接通过图形用户界面操作移动设备，为移动自动化开辟新可能。然而，现实中的移动任务通常复杂且存在多种有效解法，这与当前评估标准相矛盾：离线静态基准仅能验证单一预设的‘黄金路径’，而在线动态测试受限于真实设备的复杂性和不可复现性，两者均难以全面评估智能体能力。为此，本文提出一种新型图结构基准框架，通过建模真实设备交互中观察到的有限状态，实现动态行为的静态模拟。基于此，我们构建了专注于复杂长时序任务的ColorBench基准，支持多有效解法评估、子任务完成率统计及原子级能力分析。ColorBench包含175个任务（74个单应用，101个跨应用），平均长度超过13步，每个任务至少包含两条正确路径和若干典型错误路径，实现准动态交互。通过在多个基线上的评估，揭示现有模型的局限性，并基于实验结果提出改进方向和技术路径。代码与数据已公开。
原文摘要 · Abstract (English)
The rapid advancement of multimodal large language models has enabled agents to operate mobile devices by directly interacting with graphical user interfaces, opening new possibilities for mobile automation. However, real-world mobile tasks are often complex and allow for multiple valid solutions. This contradicts current mobile agent evaluation standards: offline static benchmarks can only validate a single predefined "golden path", while online dynamic testing is constrained by the complexity and non-reproducibility of real devices, making both approaches inadequate for comprehensively assessing agent capabilities. To bridge the gap between offline and online evaluation and enhance testing stability, this paper introduces a novel graph-structured benchmarking framework. By modeling the finite states observed during real-device interactions, it achieves static simulation of dynamic behaviors. Building on this, we develop ColorBench, a benchmark focused on complex long-horizon tasks. It supports evaluation of multiple valid solutions, subtask completion rate statistics, and atomic-level capability analysis. ColorBench contains 175 tasks (74 single-app, 101 cross-app) with an average length of over 13 steps. Each task includes at least two correct paths and several typical error paths, enabling quasi-dynamic interaction. By evaluating ColorBench across various baselines, we discover limitations of existing models and propose improvement directions and feasible technical pathways to enhance agents' performance on complex, long-horizon problems based on experimental results. Code and data are available at: https://github.com/MadeAgents/ColorBench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。