构建真实移动端任务评估框架，解决现有基准与实际应用脱节问题。
MobiFlow: Real-World Mobile Agent Benchmarking through Trajectory Fusion
- 基于多轨迹融合构建图结构，压缩状态空间并支持动态交互。
- 覆盖20个主流第三方应用，包含240项真实任务，评估指标更丰富。
- 评估结果更贴近人工判断，适合指导真实场景下的GUI模型训练。
移动代理可通过图形界面交互自主完成用户任务。然而，现有主流评估基准（如AndroidWorld）通过连接系统级Android模拟器获取资源状态作为评估信号，在真实场景中，许多第三方应用未开放系统级API来判断任务是否成功，导致评估基准与实际使用存在偏差，难以准确衡量模型性能。为此，我们提出MobiFlow，一个基于任意第三方应用任务的评估框架。通过基于多轨迹融合的高效图构建算法，MobiFlow能有效压缩状态空间，支持动态交互，并更好地贴合真实第三方应用场景。该框架涵盖20个广泛使用的第三方应用，包含240项多样化的真实任务，具备丰富评估指标。相较于AndroidWorld，MobiFlow的评估结果与人工评估更具一致性，可为未来基于GUI模型在真实负载下的训练提供有效指导。
原文摘要 · Abstract (English)
Mobile agents can autonomously complete user-assigned tasks through GUI interactions. However, existing mainstream evaluation benchmarks, such as AndroidWorld, operate by connecting to a system-level Android emulator and provide evaluation signals based on the state of system resources. In real-world mobile-agent scenarios, however, many third-party applications do not expose system-level APIs to determine whether a task has succeeded, leading to a mismatch between benchmarks and real-world usage and making it difficult to evaluate model performance accurately. To address these issues, we propose MobiFlow, an evaluation framework built on tasks drawn from arbitrary third-party applications. Using an efficient graph-construction algorithm based on multi-trajectory fusion, MobiFlow can effectively compress the state space, support dynamic interaction, and better align with real-world third-party application scenarios. MobiFlow covers 20 widely used third-party applications and comprises 240 diverse real-world tasks, with enriched evaluation metrics. Compared with AndroidWorld, MobiFlow's evaluation results show higher alignment with human assessments and can guide the training of future GUI-based models under real workloads.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。