构建首个跨设备协作任务基准，测试大模型智能体在多设备间协同能力。
DevicesWorld: Benchmarking Cross-Device Agents in Heterogeneous Environments
- 设计统一框架整合手机、电脑与物联网设备，支持端到端跨设备任务执行。
- 6140个任务中最佳模型成功率仅12.5%，多数失败因信息获取或界面操作卡顿。
- 提供可自动验证的闭环评测机制，适合研究可靠跨设备智能体的团队使用。
基于大语言模型的智能体在单一数字环境（如手机应用、桌面系统、智能家居）中的操作能力迅速提升。然而，真实用户目标往往跨越多个设备：信息可能来自手机，需在电脑上处理，结果又需在另一设备上呈现。现有基准大多聚焦单一主导执行环境，难以评估智能体在异构设备间获取与整合信息、完成具有跨设备依赖关系的端到端任务的能力。我们提出 DevicesWorld，一个大规模可执行的跨设备协作操作基准。DevicesWorld 包含 6,140 个任务，集成手机、电脑与 IoT 三类设备环境，构建统一的跨设备交互与评估框架。每个任务定义自然语言用户目标、参与设备与初始状态、可执行动作、基于规则的验证器及清理流程。通过多阶段构建与质量控制流程，确保任务贴近真实用户需求，同时允许从设备状态和生成文件中自动验证最终结果。我们在固定评估集上对五个前沿大模型智能体系统进行评测，所有方法成功率均较低，最优者仅达 12.5%。在失败案例中，约 28.7% 满足至少一项评分条件但仍未完成全任务。轨迹分析显示，智能体常在信息获取或界面操作中卡住，混淆源设备与输出设备，或在所有条件未同时满足前提前终止。DevicesWorld 将跨设备协作操作转化为可执行、可复现且具诊断价值的评估问题，推动可靠跨设备智能体的研究。
原文摘要 · Abstract (English)
LLM-based agents have rapidly improved at operating individual digital environments such as mobile applications, desktop systems, and smart homes. However, real-world user goals often span multiple devices: information may come from a phone, be processed on a desktop, and the result may need to appear on another device. Most existing benchmarks center on a single dominant execution environment, making it difficult to evaluate whether agents can acquire and integrate information across heterogeneous devices and complete end-to-end tasks with cross-device dependencies. We introduce DevicesWorld, a large-scale executable benchmark for cross-device collaborative operation. DevicesWorld contains 6,140 tasks and integrates three classes of device environments -- mobile, desktop, and IoT -- into a unified cross-device interaction and evaluation framework. Each task defines a natural-language user goal, participating devices and initial states, executable actions, rule-based verifiers, and a cleanup procedure. A multi-stage construction and quality-control pipeline keeps tasks close to realistic user needs while allowing final outcomes to be automatically verified from device states and generated files. We evaluate five frontier LLM-agent systems on a fixed evaluation set. All methods achieve low success rates, with the best reaching only 12.5%. Among failed runs, about 28.7% satisfy at least one scoring condition yet still fail the full task. Trajectories show that agents become stuck acquiring information or manipulating interfaces, confuse source and output devices, or terminate before all conditions are jointly satisfied. DevicesWorld turns cross-device collaborative operation into an executable, reproducible, and diagnostically useful evaluation problem for research on reliable cross-device agents.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。