用3D高斯点云增强多模态检索,实现零样本机器人抓取。
Zero-Shot Robotic Manipulation via 3D Gaussian Splatting-Enhanced Multimodal Retrieval-Augmented Generation
- 构建多源知识库,通过层级检索匹配任务相关物体原型。
- 引入3D高斯点云对齐技术,提升参考物体与目标的几何匹配精度。
- 在30类家用物品上成功率提升7.76%,兼顾语义理解与几何执行。
现有端到端机器人抓取方法因数据有限和可解释性差,难以泛化至未见物体或任务。尽管多模态大语言模型具备强常识推理能力,但在姿态预测所需的几何空间理解方面表现不足。本文提出RobMRAG框架,基于3D高斯点云增强的多模态检索增强生成技术,实现零样本机器人操作。构建包含物体接触帧、任务完成帧和姿态参数的多源知识库。推理时,分层多模态检索模块采用三优先级混合策略,先定位任务相关的物体原型,再根据像素级相似性和实例匹配距离(IMD)选择几何最接近的参考样本。进一步引入基于3D高斯点云的3D感知姿态精修模块,将参考物体姿态对齐至目标物体3D空间。对齐结果重投影至图像平面后输入多模态大语言模型,以生成最终姿态参数。大量实验表明,在包含30类家用物品的测试集上,本方法相较最佳零样本基线成功率提升7.76%,较最先进的监督学习基线提升6.54%。结果验证了RobMRAG有效弥合高层语义推理与低层几何执行之间的差距,使机器人系统在保持可解释性的同时泛化至未见物体。
原文摘要 · Abstract (English)
Existing end-to-end approaches of robotic manipulation often lack generalization to unseen objects or tasks due to limited data and poor interpretability. While recent Multimodal Large Language Models (MLLMs) demonstrate strong commonsense reasoning, they struggle with geometric and spatial understanding required for pose prediction. In this paper, we propose RobMRAG, a 3D Gaussian Splatting-Enhanced Multimodal Retrieval-Augmented Generation (MRAG) framework for zero-shot robotic manipulation. Specifically, we construct a multi-source manipulation knowledge base containing object contact frames, task completion frames, and pose parameters. During inference, a Hierarchical Multimodal Retrieval module first employs a three-priority hybrid retrieval strategy to find task-relevant object prototypes, then selects the geometrically closest reference example based on pixel-level similarity and Instance Matching Distance (IMD). We further introduce a 3D-Aware Pose Refinement module based on 3D Gaussian Splatting into the MRAG framework, which aligns the pose of the reference object to the target object in 3D space. The aligned results are reprojected onto the image plane and used as input to the MLLM to enhance the generation of the final pose parameters. Extensive experiments show that on a test set containing 30 categories of household objects, our method improves the success rate by 7.76% compared to the best-performing zero-shot baseline under the same setting, and by 6.54% compared to the state-of-the-art supervised baseline. Our results validate that RobMRAG effectively bridges the gap between high-level semantic reasoning and low-level geometric execution, enabling robotic systems that generalize to unseen objects while remaining inherently interpretable.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。