用视觉语言模型提升机器人抓取的自然语言理解能力
HiFi-CS: Towards Open Vocabulary Visual Grounding For Robotic Grasping Using Vision-Language Models
- 通过分层特征调制融合图像与文本嵌入，增强复杂语义理解
- 在15个桌面上实现90.33%的视觉定位准确率，性能超越基线
- 轻量级设计适合实际机器人部署，支持开放词汇识别
能通过自然语言与人类交互的机器人可拓展众多应用场景，如指代抓取合成（RGS）。给定文本查询，RGS需确定机器人工作空间中目标物体的稳定抓取姿态。该任务包含视觉定位与抓取姿态估计两步。近期研究利用强大的视觉语言模型（VLM）在真实机器人执行中对自由流动的自然语言进行视觉定位。然而，缺乏在复杂、杂乱环境中多个相同物体实例下的对比验证。本文提出HiFi-CS，采用分层特征调制（FiLM）融合图像与文本嵌入，提升复杂属性丰富文本查询下的视觉定位能力。视觉定位将2D/3D空间中的物体与自然语言输入关联，分为封闭词汇与开放词汇两种场景。HiFi-CS结合轻量解码器与冻结的VLM，在封闭词汇设置下表现优于竞争基线，且模型大小仅为后者的1/100。本模型可有效引导开放集检测器如GroundedSAM，提升开放词汇性能。通过7自由度机械臂在真实世界中验证，15个桌面上实现90.33%的视觉定位准确率。代码已开源：https://github.com/vineet2104/hifics
原文摘要 · Abstract (English)
Robots interacting with humans through natural language can unlock numerous applications such as Referring Grasp Synthesis (RGS). Given a text query, RGS determines a stable grasp pose to manipulate the referred object in the robot's workspace. RGS comprises two steps: visual grounding and grasp pose estimation. Recent studies leverage powerful Vision-Language Models (VLMs) for visually grounding free-flowing natural language in real-world robotic execution. However, comparisons in complex, cluttered environments with multiple instances of the same object are lacking. This paper introduces HiFi-CS, featuring hierarchical application of Featurewise Linear Modulation (FiLM) to fuse image and text embeddings, enhancing visual grounding for complex attribute rich text queries encountered in robotic grasping. Visual grounding associates an object in 2D/3D space with natural language input and is studied in two scenarios: Closed and Open Vocabulary. HiFi-CS features a lightweight decoder combined with a frozen VLM and outperforms competitive baselines in closed vocabulary settings while being 100x smaller in size. Our model can effectively guide open-set object detectors like GroundedSAM to enhance open-vocabulary performance. We validate our approach through real-world RGS experiments using a 7-DOF robotic arm, achieving 90.33\% visual grounding accuracy in 15 tabletop scenes. Our codebase is provided here: https://github.com/vineet2104/hifics
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。