用大模型零样本实现智能抓取,不训练也能懂物体怎么抓。
ORACLE-Grasp: Zero-Shot Affordance-Aligned Robotic Grasping using Large Multimodal Models
- 用大模型提取物体语义和可抓区域,分步推理抓取策略。
- 在真实场景中抓取成功率高,接近人类标注水平。
- 无需训练或建模,适合新物品和复杂环境的机器人使用。
在家庭、医院、仓库等非结构化环境中抓取未知物体是服务机器人的关键挑战,需兼具语义理解与空间推理能力。传统方法依赖密集数据集或详细几何建模,需大量数据采集且泛化能力差。本文提出ORACLE-Grasp,一种零样本框架,利用大型多模态模型(LMMs)作为语义“圣谕”,引导与功能对齐的抓取选择,无需任务特定训练或人工输入。系统将抓取预测重构为结构化的迭代决策过程,采用双提示工具调用策略:第一提示提取物体高层次语义,第二提示识别与功能一致的可抓区域。为克服LMM的空间局限,系统将图像离散化为候选区域,逐个推理生成类人且情境敏感的抓取建议。深度信息可用时进行优化以提升可靠性,早期停止机制增强计算效率。我们在包含日常物品和AI生成物体的RGB与RGB-D图像上评估该方法,结果表明其生成的抓取方案物理可行、语义合理,与人类标注高度一致,在真实抓取任务中取得高成功率。研究揭示了LMM在实现自主服务机器人灵活、泛化抓取策略中的潜力,无需对象专属模型或大量训练。
原文摘要 · Abstract (English)
Grasping unknown objects in unstructured environments is a critical challenge for service robots, which must operate in dynamic, real-world settings such as homes, hospitals, and warehouses. Success in these environments requires both semantic understanding and spatial reasoning. Traditional methods often rely on dense training datasets or detailed geometric modeling, which demand extensive data collection and do not generalize well to novel objects or affordances. We present ORACLE-Grasp, a zero-shot framework that leverages Large Multimodal Models (LMMs) as semantic oracles to guide affordance-aligned grasp selection, without requiring task-specific training or manual input. The system reformulates grasp prediction as a structured, iterative decision process, using a dual-prompt tool-calling strategy: the first prompt extracts high-level object semantics, while the second identifies graspable regions aligned with the object's function. To address the spatial limitations of LMMs, ORACLE-Grasp discretizes the image into candidate regions and reasons over them to produce human-like and context-sensitive grasp suggestions. A depth-based refinement step improves grasp reliability when available, and an early stopping mechanism enhances computational efficiency. We evaluate ORACLE-Grasp on a diverse set of RGB and RGB-D images featuring both everyday and AI-generated objects. The results show that our method produces physically feasible and semantically appropriate grasps that align closely with human annotations, achieving high success rates in real-world pick-up tasks. Our findings highlight the potential of LMMs for enabling flexible and generalizable grasping strategies in autonomous service robots, eliminating the need for object-specific models or extensive training.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。