用视觉语言模型+扩散动作控制器,实现90%以上抓取成功率的通用灵巧操作。
DexGraspVLA: A Vision-Language-Action Framework Towards General Dexterous Grasping

- 分层框架:上层用预训练视觉语言模型规划,下层用扩散模型生成动作。
- 在数千个未见杂乱场景中达到90%以上抓取成功率,显著优于传统方法。
- 支持长序列指令、抗干扰和失败恢复,适合真实复杂环境中的机器人应用。
灵巧抓取是机器人领域的基础挑战。现有方法多依赖单物体或受限环境假设,泛化能力有限。本文提出DexGraspVLA,一种分层式视觉-语言-动作框架,用于语言引导的通用灵巧抓取及更广泛应用。该框架利用预训练视觉语言模型作为高层规划器,学习基于扩散模型的低层动作控制器。核心思想是通过基础模型将多样化的语言与视觉输入迭代转换为领域不变表示,从而缓解领域偏移,使模仿学习得以有效应用。实验表明,该方法在数千个未见的杂乱场景中实现超过90%的灵巧抓取成功率。实证分析验证了模型内部行为在环境变化下的稳定性。DexGraspVLA首次同时实现自由形式长时序指令执行、对对抗性物体和人为干扰的鲁棒性以及失败恢复能力。扩展至非抓握操作进一步证明其通用性。
原文摘要 · Abstract (English)
Dexterous grasping remains a fundamental yet challenging problem in robotics. A general-purpose robot must be capable of grasping diverse objects in arbitrary scenarios. However, existing research typically relies on restrictive assumptions, such as single-object settings or limited environments, showing constrained generalization. We present DexGraspVLA, a hierarchical framework for robust generalization in language-guided general dexterous grasping and beyond. It utilizes a pre-trained Vision-Language model as the high-level planner and learns a diffusion-based low-level Action controller. The key insight to achieve generalization lies in iteratively transforming diverse language and visual inputs into domain-invariant representations via foundation models, where imitation learning can be effectively applied due to the alleviation of domain shift. Notably, our method achieves a 90+% dexterous grasping success rate under thousands of challenging unseen cluttered scenes. Empirical analysis confirms the consistency of internal model behavior across environmental variations, validating our design. DexGraspVLA also, for the first time, simultaneously demonstrates free-form long-horizon prompt execution, robustness to adversarial objects and human disturbance, and failure recovery. Extended application to nonprehensile grasping further proves its generality. Project website: https://dexgraspvla.github.io.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。