一人操控多机械臂,智能助手分担任务,提升协作效率。
HATS: A Human-Agent Teleoperation System for Multi-Arm Data Collection

- 人机协同控制:主臂由人直接操作,辅臂由无需训练的智能体执行子任务。
- 单人操作成功率接近双人专家团队,数据收集效率高。
- 支持语音指令干预,适合复杂多臂协作场景的数据采集。
许多现实中的操作场景,如复杂协作任务和大空间作业,需要多于两个机械臂的协同。因此,需有效的多臂遥操作系统来收集训练协调多臂操作策略的示范数据。然而,现有遥操作框架主要集中在单操作员或多人操作设置,面临单操作员认知负荷与多人协作成本之间的实际权衡。为解决此问题,我们提出了 HATS——一种人-智能体遥操作系统,使单一人类操作员在基于多模态大语言模型(MLLM)的智能体协助下,完成多臂操作任务的数据采集。该系统将控制空间解耦:两个主臂由人类直接遥操作,两个辅助臂则由无需训练的智能体负责处理子任务。此外,操作员可在执行过程中通过语音指令防止碰撞并纠正辅助臂行为。大量实验表明,HATS 的数据采集效率和成功率可媲美专家级双人团队。下游策略评估进一步验证了所采集数据的有效性与高质量。
原文摘要 · Abstract (English)
Many real-world manipulation scenarios, such as handling complex collaborative tasks and dealing with large workspaces, require coordination of more than two robotic arms. Consequently, an effective multi-arm teleoperation system is required to collect demonstrations for training coordinated multi-arm manipulation policies. However, existing teleoperation frameworks mainly focus on single-operator or multi-operator setups, facing a practical trade-off between the cognitive load placed on a single operator and the coordination cost incurred by multiple operators. To address this problem, we introduce HATS, a human-agent teleoperation system that enables a single human operator, assisted by an MLLM-based agent, to collect data for multi-arm manipulation tasks. Our system decouples the control space: two primary arms are directly teleoperated by the human, while two assistive arms are controlled by a training-free agent that handles sub-tasks. In addition, the human operator can use voice commands to prevent collisions and correct assistive arm behaviors during execution. Extensive evaluations demonstrate that HATS achieves data collection efficiency and success rates comparable to expert dual-human teams. Moreover, downstream policy evaluations demonstrate the efficacy and quality of the data collected through HATS.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。