用合成数据训练的机器人抓取模型，能听懂自然语言指令并准确抓取物品。
GraspMolmo: Generalizable Task-Oriented Grasping via Large-Scale Synthetic Data Generation
- 基于37.9万条合成数据，用视觉语言模型训练抓取策略。
- 真实场景下复杂任务成功率70%，超过现有方法一倍。
- 零样本适配新指令和双手操作，适合智能机器人研发者。
我们提出GraspMolmo，一个可泛化的开放词汇任务导向抓取（TOG）模型。该模型根据自然语言指令与单张RGB-D图像预测语义合理且稳定的抓取姿势。例如，当指令为“给我倒点茶”时，模型会选择茶壶把手而非壶身进行抓取。不同于以往受限于小规模数据、简单语言和无杂乱环境的方法，GraspMolmo在全新大规模合成数据集PRISM上进行训练，该数据集包含37.9万条样本，涵盖杂乱环境和多样真实的任务描述。我们在该数据上微调Molmo视觉-语言模型，使GraspMolmo能泛化至未见的开放词汇指令与物体。在挑战性的真实世界评估中，其复杂任务预测成功率高达70%，显著优于次优方案的35%。此外，模型还成功实现零样本预测语义正确的双手抓取。我们公开了合成数据集、代码、模型及评测基准，以加速任务语义机器人操作研究，相关资源与视频详见https://abhaybd.github.io/GraspMolmo/。
原文摘要 · Abstract (English)
We present GrasMolmo, a generalizable open-vocabulary task-oriented grasping (TOG) model. GraspMolmo predicts semantically appropriate, stable grasps conditioned on a natural language instruction and a single RGB-D frame. For instance, given "pour me some tea", GraspMolmo selects a grasp on a teapot handle rather than its body. Unlike prior TOG methods, which are limited by small datasets, simplistic language, and uncluttered scenes, GraspMolmo learns from PRISM, a novel large-scale synthetic dataset of 379k samples featuring cluttered environments and diverse, realistic task descriptions. We fine-tune the Molmo visual-language model on this data, enabling GraspMolmo to generalize to novel open-vocabulary instructions and objects. In challenging real-world evaluations, GraspMolmo achieves state-of-the-art results, with a 70% prediction success on complex tasks, compared to the 35% achieved by the next best alternative. GraspMolmo also successfully demonstrates the ability to predict semantically correct bimanual grasps zero-shot. We release our synthetic dataset, code, model, and benchmarks to accelerate research in task-semantic robotic manipulation, which, along with videos, are available at https://abhaybd.github.io/GraspMolmo/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。