用视觉语言模型零样本检测抓取,无需训练数据即可泛化到新物体。
VLAD-Grasp: Zero-shot Grasp Detection via Vision-Language Models
- 通过语言模型生成带抓取轴的虚拟图像,编码抓取方向。
- 结合深度与分割信息重建3D抓取姿态,实现零样本推理。
- 无需标注数据,在真实机器人上成功抓取未知物体。
机器人抓取是实现自主操作的基础能力,通常存在无限解。当前先进方法依赖大规模带有专家标注可行抓取的数据集,但数据标注困难,导致学习方法受限于数据覆盖范围,且需重新训练以应对新物体。为此,我们提出VLAD-Grasp:一种基于视觉语言模型的零样本抓取检测方法。该方法首先利用大型视觉语言模型生成一个目标图像,其中虚拟圆柱体与物体几何相交,显式编码对称抓取轴;然后预测深度和分割,将生成图像提升至3D空间;最后通过主成分分析和无对应点优化,对齐生成与观测物体点云,恢复可执行抓取位姿。与以往工作不同,本方法无需训练且不依赖标注抓取数据集,在Cornell和Jacquard数据集上性能媲美最先进方法。此外,我们在Franka Research 3机器人上验证了其在真实物体上的零样本泛化能力,表明视觉语言模型可作为机器人操作的强大先验知识。
原文摘要 · Abstract (English)
Robotic grasping is a fundamental capability for enabling autonomous manipulation, with usually infinite solutions. State-of-the-art approaches for grasping rely on learning from large-scale datasets comprising expert annotations of feasible grasps. Curating such datasets is challenging, and hence, learning-based methods are limited by the solution coverage of the dataset, and require retraining to handle novel objects. Towards this, we present VLAD-Grasp, a Vision-Language model Assisted zero-shot approach for Detecting Grasps. Our method (1) prompts a large vision-language model to generate a goal image where a virtual cylindrical proxy intersects the object's geometry, explicitly encoding an antipodal grasp axis in image space, then (2) predicts depth and segmentation to lift this generated image into 3D, and (3) aligns generated and observed object point clouds via principal components and correspondence-free optimization to recover an executable grasp pose. Unlike prior work, our approach is training-free and does not require curated grasp datasets, while achieving performance competitive with the state-of-the-art methods on the Cornell and Jacquard datasets. Furthermore, we demonstrate zero-shot generalization to real-world objects on a Franka Research 3 robot, highlighting vision-language models as powerful priors for robotic manipulation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。