用语言指导机器人在复杂场景中抓取，支持多种机械臂形态。
SeededGrasp: Language-Guided Grasping in Complex Scenes with Multiple Embodiments
- 先让语言模型选抓取起点，再由轻量模型生成具体动作。
- 仿真成功率达72%，真实世界达78%，优于现有方法。
- 首个支持多机械臂的桌面上抓取数据集，含250万次抓取。
复杂场景中的实用机器人抓取需要3D空间推理与任务需求的精准对齐。视觉语言模型（VLM）可通过自然语言指定这些需求，但现有方法要么直接用VLM预测抓取动作而空间感知不足，要么将VLM与抓取模型联合训练，需大量数据和计算资源。为此，我们提出SeededGrasp，一种数据高效的框架：让VLM预测一个种子点作为后续轻量级抓取生成模型的条件输入。该架构将高层语义推理与低层几何执行解耦，实现多形态机械臂支持，同时避免昂贵的端到端训练。为支持此类模型训练，我们发布了首个包含超过250万次抓取的多形态桌面抓取数据集。实验表明，本方法在仿真中达到72%成功率，在真实世界实验中达78%。项目主页提供数据与代码：https://uoft-isl.github.io/seeded-grasp/
原文摘要 · Abstract (English)
Practical robotic grasping in complex scenes requires both 3D spatial reasoning and alignment with task-specific requirements. Vision-language models (VLMs) offer a natural way to specify these requirements using language, but existing approaches either use a VLM to predict the grasp directly with limited spatial awareness, or train the VLM together with the grasping model, which requires significantly more data and compute. These limitations impede performance and have prevented scaling to multiple embodiments in complex scenes. We address this by proposing SeededGrasp, a novel data-efficient framework that enables a VLM to predict a seed point to be used as conditioning for a subsequent lightweight grasp-generation model. Our architecture decouples high-level semantic reasoning from low-level geometric execution, enabling multi-embodiment support while bypassing the need for expensive end-to-end training. To enable training such models, we release the first multi-embodiment tabletop grasping dataset comprising over 2.5M grasps in cluttered scenes. Experimental results demonstrate that our approach outperforms existing baselines, achieving 72% success in simulation and 78% in real-world grasping experiments. See our project site for data and code: https://uoft-isl.github.io/seeded-grasp/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。