机器人读说明书操作家电,提升任务成功率。
Robot Operation of Home Appliances by Reading User Manuals
- 用大模型从说明书提取结构化控制策略
- 在真实设备上实现90%以上任务成功率
- 适合研究家用机器人自主操作的学者
操作家用电器是辅助家庭机器人的重要能力。本文提出ApBot系统,让机器人通过阅读用户手册来操作新型家电。该系统面临三大挑战:(i) 从非结构化的文本描述中推断目标相关的部分动作策略;(ii) 将策略与物理设备上的控制面板进行视觉对齐;(iii) 在多步执行中可靠运行,克服累积误差。为此,ApBot借助大视觉语言模型(VLM)从手册构建家电的结构化符号模型,通过视觉方式将符号动作映射到控制面板元素,并基于视觉反馈动态更新模型。实验表明,在多种模拟和真实家电上,与直接使用最先进的大VLM作为控制策略相比,ApBot在任务成功率上取得一致且统计显著的提升。结果表明,结构化内部表征在复杂家电的鲁棒操作中起关键作用。
原文摘要 · Abstract (English)
Operating home appliances, among the most common tools in every household, is a critical capability for assistive home robots. This paper presents ApBot, a robot system that operates novel household appliances by "reading" their user manuals. ApBot faces multiple challenges: (i) infer goal-conditioned partial policies from their unstructured, textual descriptions in a user manual document, (ii) ground the policies to the appliance in the physical world, and (iii) execute the policies reliably over potentially many steps, despite compounding errors. To tackle these challenges, ApBot constructs a structured, symbolic model of an appliance from its manual, with the help of a large vision-language model (VLM). It grounds the symbolic actions visually to control panel elements. Finally, ApBot closes the loop by updating the model based on visual feedback. Our experiments show that across a wide range of simulated and real-world appliances, ApBot achieves consistent and statistically significant improvements in task success rate, compared with state-of-the-art large VLMs used directly as control policies. These results suggest that a structured internal representations plays an important role in robust robot operation of home appliances, especially, complex ones.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。