让机器人通过理解物体可操作性，零样本完成复杂指令下的移动操作任务。
Affordance RAG: Hierarchical Multimodal Retrieval with Affordance-Aware Embodied Memory for Mobile Manipulation
- 构建具可操作性的具身记忆库，分层检索目标物体。
- 在真实环境中实现85%的任务成功率，优于现有方法。
- 适合需要灵活执行自然语言指令的移动机械臂场景。
本文针对开放词汇的移动操作任务，即机器人需根据自由形式的自然语言指令将多种物体搬运至指定位置。该任务难点在于理解视觉语义与操作可行性。为此，我们提出Affordance RAG，一种零样本的层次化多模态检索框架，利用预先探索图像构建具可操作性的具身记忆。模型基于区域和视觉语义检索候选目标，并通过可操作性评分重排序，使机器人能识别现实环境中可执行的操作选项。在大规模室内环境中，该方法在检索性能上超越现有方法；真实世界实验中，机器人基于自由形式指令完成移动操作，任务成功率达到85%，在检索性能与整体任务成功率上均优于现有方法。
原文摘要 · Abstract (English)
In this study, we address the problem of open-vocabulary mobile manipulation, where a robot is required to carry a wide range of objects to receptacles based on free-form natural language instructions. This task is challenging, as it involves understanding visual semantics and the affordance of manipulation actions. To tackle these challenges, we propose Affordance RAG, a zero-shot hierarchical multimodal retrieval framework that constructs Affordance-Aware Embodied Memory from pre-explored images. The model retrieves candidate targets based on regional and visual semantics and reranks them with affordance scores, allowing the robot to identify manipulation options that are likely to be executable in real-world environments. Our method outperformed existing approaches in retrieval performance for mobile manipulation instruction in large-scale indoor environments. Furthermore, in real-world experiments where the robot performed mobile manipulation in indoor environments based on free-form instructions, the proposed method achieved a task success rate of 85%, outperforming existing methods in both retrieval performance and overall task success.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。