用视觉推理引导主动感知，让大模型学会判断物体重量等非视觉属性。
Discovering Object Attributes by Prompting Large Language Models with Perception-Action APIs
- 通过视觉语言模型与大模型协同，生成控制机器人主动探测的程序。
- 在奇数出列数据集上，对物体位置、大小、重量的判断准确率优于纯视觉模型。
- 适用于需要物理交互的智能机器人场景，如家庭服务或自动化检测。
如何将自然语言与物理实体关联是当前研究热点。尽管视觉语言模型（VLMs）能将语言指令映射到视觉信息，但在判断非视觉属性（如物体重量）方面表现不佳。本文的核心洞察是：通过视觉推理引导的主动感知，可有效识别非视觉属性。为此，我们提出一个由VLM和大语言模型（LLM）驱动的感知-动作API，包含一组机器人控制函数。当接收到自然语言查询时，LLM会生成一段程序，驱动机器人主动获取信息以识别属性。离线测试在Odd-One-Out数据集上显示，该框架在相对位置、大小和重量等属性检测上优于纯VLM。在线测试在AI2-THOR仿真环境及真实家居场景中进行，使用DJI RoboMaster EP机器人完成演示，验证了方法的有效性。
原文摘要 · Abstract (English)
There has been a lot of interest in grounding natural language to physical entities through visual context. While Vision Language Models (VLMs) can ground linguistic instructions to visual sensory information, they struggle with grounding non-visual attributes, like the weight of an object. Our key insight is that non-visual attribute detection can be effectively achieved by active perception guided by visual reasoning. To this end, we present a perception-action API that consists of VLMs and Large Language Models (LLMs) as backbones, together with a set of robot control functions. When prompted with this API and a natural language query, an LLM generates a program to actively identify attributes given an input image. Offline testing on the Odd-One-Out dataset demonstrates that our framework outperforms vanilla VLMs in detecting attributes like relative object location, size, and weight. Online testing in realistic household scenes on AI2-THOR and a real robot demonstration on a DJI RoboMaster EP robot highlight the efficacy of our approach.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。