用机械臂直接操作手机，不依赖ADB，跨平台通用。
See-Control: A Multimodal Agent Framework for Smartphone Interaction with a Robotic Arm
- 通过机械臂物理操作手机，无需ADB或系统权限
- 构建155项任务的评测基准与标注数据集
- 适合家庭机器人研究者，推动真实场景交互
多模态大语言模型（MLLM）已用于智能手机智能代理。但现有方法依赖Android调试桥（ADB）进行数据传输和指令执行，仅适用于安卓设备。本文提出新型具身手机操作（ESO）任务，并构建See-Control框架，实现通过低自由度机械臂直接物理操作手机，提供无平台限制的解决方案。该框架包含三部分：(1) 包含155个任务及评估指标的ESO基准；(2) 基于MLLM的具身代理，无需ADB或系统后端访问即可生成机械臂控制指令；(3) 丰富标注的操作过程数据集，为后续研究提供资源。该工作弥合了数字代理与物理世界之间的鸿沟，推动家用机器人在真实环境中完成依赖手机的任务。
原文摘要 · Abstract (English)
Recent advances in Multimodal Large Language Models (MLLMs) have enabled their use as intelligent agents for smartphone operation. However, existing methods depend on the Android Debug Bridge (ADB) for data transmission and action execution, limiting their applicability to Android devices. In this work, we introduce the novel Embodied Smartphone Operation (ESO) task and present See-Control, a framework that enables smartphone operation via direct physical interaction with a low-DoF robotic arm, offering a platform-agnostic solution. See-Control comprises three key components: (1) an ESO benchmark with 155 tasks and corresponding evaluation metrics; (2) an MLLM-based embodied agent that generates robotic control commands without requiring ADB or system back-end access; and (3) a richly annotated dataset of operation episodes, offering valuable resources for future research. By bridging the gap between digital agents and the physical world, See-Control provides a concrete step toward enabling home robots to perform smartphone-dependent tasks in realistic environments.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。