arXiv:2512.19453cs.RO2025-12被引 1

用抽象元动作+检索增强生成,让机器人更懂复杂日常任务的规划。

MaP-AVR: A Meta-Action Planner for Agents Leveraging Vision Language Models and Retrieval-Augmented Generation

  • 将任务分解为由动作、末端执行器状态、环境关系组成的元动作。
  • 在OmniGibson平台上使用GPT-4o实现任务成功率优于当前最优方法。
  • 通过人类标注示范库自增数据,持续提升规划多样性与泛化能力。

面向复杂日常任务的具身机器人智能系统依赖任务规划器来理解并分解高层任务。现有研究多聚焦于通过微调或思维链提示提升大语言模型(LLM)/视觉语言模型(VLM)的任务理解能力,本文认为定义可规划技能集同样关键。为应对真实环境复杂性,技能集需具备强泛化能力,而更抽象的表达形式更具通用性。因此,我们提出将规划结果抽象为一组元动作,每个元动作包含三部分:{移动/旋转、末端执行器状态变化、与环境的关系}。该抽象替代了人类中心概念(如抓取、推动),转而基于机器人的内在功能,使规划输出与机器人实际可执行动作完全对齐。为进一步确保LLM/VLM准确生成所需元动作格式,引入检索增强生成(RAG)技术,利用人类标注的规划示范数据库实现上下文学习。随着系统成功完成更多任务,数据库自动扩充,持续支持规划多样性。元动作集合及其与RAG的融合是本规划器MaP-AVR(Meta-Action Planner for Agents with VLM and RAG)的两项核心创新。实验以GPT-4o为预训练模型,OmniGibson为机器人平台验证有效性,结果表明其性能优于当前最先进方法。

原文摘要 · Abstract (English)

Embodied robotic AI systems designed to manage complex daily tasks rely on a task planner to understand and decompose high-level tasks. While most research focuses on enhancing the task-understanding abilities of LLMs/VLMs through fine-tuning or chain-of-thought prompting, this paper argues that defining the planned skill set is equally crucial. To handle the complexity of daily environments, the skill set should possess a high degree of generalization ability. Empirically, more abstract expressions tend to be more generalizable. Therefore, we propose to abstract the planned result as a set of meta-actions. Each meta-action comprises three components: {move/rotate, end-effector status change, relationship with the environment}. This abstraction replaces human-centric concepts, such as grasping or pushing, with the robot's intrinsic functionalities. As a result, the planned outcomes align seamlessly with the complete range of actions that the robot is capable of performing. Furthermore, to ensure that the LLM/VLM accurately produces the desired meta-action format, we employ the Retrieval-Augmented Generation (RAG) technique, which leverages a database of human-annotated planning demonstrations to facilitate in-context learning. As the system successfully completes more tasks, the database will self-augment to continue supporting diversity. The meta-action set and its integration with RAG are two novel contributions of our planner, denoted as MaP-AVR, the meta-action planner for agents composed of VLM and RAG. To validate its efficacy, we design experiments using GPT-4o as the pre-trained LLM/VLM model and OmniGibson as our robotic platform. Our approach demonstrates promising performance compared to the current state-of-the-art method. Project page: https://map-avr.github.io/.

机器人规划视觉语言模型检索增强元动作

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。