用视觉语言模型指导抓取语义，实现更智能的灵巧抓取生成。
OmniDexVLG: Learning Dexterous Grasp Generation from Vision Language Model-Guided Grasp Semantics, Taxonomy and Functional Affordance
- 融合抓取分类、接触语义与功能属性，统一建模多重语义维度。
- 在仿真与真实物体上，抓取多样性与语义一致性显著优于现有方法。
- 适合需要自然语言控制灵巧抓取的机器人研究与应用。
灵巧抓取生成旨在产生符合任务需求且具备人类可理解语义的抓取姿态。然而，由于缺乏对抓取分类、接触语义和功能可用性等多维语义的统一建模，实现语义可控的灵巧抓取合成仍面临巨大挑战。为此，我们提出OmniDexVLG，一个具备多模态语义感知能力的抓取生成框架，可在语言与视觉联合引导下生成结构多样且语义一致的灵巧抓取。该方法首先构建OmniDexDataGen，通过分类引导采样、功能可用性接触点采样、分类感知的差分力闭合抓取采样及物理优化验证，系统覆盖多种抓取类型。进一步引入OmniDexReasoner，利用多智能体协作、检索增强生成与思维链推理，推断抓取相关语义并生成高质量标注，使语言指令与任务意图对齐。基于此，我们构建统一的视觉语言抓取生成模型，显式融合抓取分类、接触结构与功能可用性语义，实现从自然语言指令中精细控制抓取生成。大量仿真实验与真实世界抓取测试表明，本方法在抓取多样性、接触语义多样性、功能可用性多样性与语义一致性方面均显著优于当前最优方法。
原文摘要 · Abstract (English)
Dexterous grasp generation aims to produce grasp poses that align with task requirements and human interpretable grasp semantics. However, achieving semantically controllable dexterous grasp synthesis remains highly challenging due to the lack of unified modeling of multiple semantic dimensions, including grasp taxonomy, contact semantics, and functional affordance. To address these limitations, we present OmniDexVLG, a multimodal, semantics aware grasp generation framework capable of producing structurally diverse and semantically coherent dexterous grasps under joint language and visual guidance. Our approach begins with OmniDexDataGen, a semantic rich dexterous grasp dataset generation pipeline that integrates grasp taxonomy guided configuration sampling, functional affordance contact point sampling, taxonomy aware differential force closure grasp sampling, and physics based optimization and validation, enabling systematic coverage of diverse grasp types. We further introduce OmniDexReasoner, a multimodal grasp type semantic reasoning module that leverages multi agent collaboration, retrieval augmented generation, and chain of thought reasoning to infer grasp related semantics and generate high quality annotations that align language instructions with task specific grasp intent. Building upon these components, we develop a unified Vision Language Grasping generation model that explicitly incorporates grasp taxonomy, contact structure, and functional affordance semantics, enabling fine grained control over grasp synthesis from natural language instructions. Extensive experiments in simulation and real world object grasping and ablation studies demonstrate that our method substantially outperforms state of the art approaches in terms of grasp diversity, contact semantic diversity, functional affordance diversity, and semantic consistency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。