用视觉语言模型闭环迭代,让机器人精准按文字指令摆放物体。
Text-Guided 6D Object Pose Rearrangement via Closed-Loop VLM Agents

- 通过多视角推理和坐标系可视化,让模型逐步优化物体姿态。
- 无需微调,在多个数据集上显著优于现有方法,成功率超90%。
- 适合想用现成大模型实现复杂操作的机器人研究者。
视觉语言模型(VLM)具备强大的视觉推理能力,但在3D理解方面仍存在挑战,尤其难以推断出与文本一致的目标物体6D位姿。本文发现,通过推理时技术与迭代推理,可显著提升性能。给定一个由RGB-D图像或3D网格组成的3D场景及一段描述目标状态变化的文本指令,我们重复以下闭环流程:观察当前场景;评估其是否符合指令;提出目标物体的姿态更新;执行更新;渲染新场景。通过这一过程,VLM充当智能代理。我们引入三种关键推理时技术:(i) 支持性视图选择的多视角推理,(ii) 以物体为中心的坐标系可视化,(iii) 单轴旋转预测。该方法无需额外微调或新增模块,即可在文本引导的6D位姿预测任务中超越已有方法,且在闭源与开源VLM上均表现稳定。结合简单运动规划,其机器人操作成功率高于近期视觉-语言-动作模型(VLAs)。消融实验验证了每项技术的必要性。
原文摘要 · Abstract (English)
Vision-Language Models (VLMs) exhibit strong visual reasoning capabilities, yet they still struggle with 3D understanding. In particular, VLMs often fail to infer a text-consistent goal 6D pose of a target object in a 3D scene. However, we find that with some inference-time techniques and iterative reasoning, VLMs can achieve dramatic performance gains. Concretely, given a 3D scene represented by an RGB-D image (or a compositional scene of 3D meshes) and a text instruction specifying a desired state change, we repeat the following loop: observe the current scene; evaluate whether it is faithful to the instruction; propose a pose update for the target object; apply the update; and render the updated scene. Through this closed-loop interaction, the VLM effectively acts as an agent. We further introduce three inference-time techniques that are essential to this closed-loop process: (i) multi-view reasoning with supporting view selection, (ii) object-centered coordinate system visualization, and (iii) single-axis rotation prediction. Without any additional fine-tuning or new modules, our approach surpasses prior methods at predicting the text-guided goal 6D pose of the target object. It works consistently across both closed-source and open-source VLMs. Moreover, when combining our 6D pose prediction with simple robot motion planning, it enables more successful robot manipulation than recent Vision-Language-Action models (VLAs). Finally, we conduct an ablation study to demonstrate the necessity of each proposed technique.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。