让机器人更懂人类行为，用轻量模块实现视觉与语言的双向推理。
Lightweight Visual Reasoning for Socially-Aware Robots
- 设计轻量级语言-视觉反馈模块，让模型可重读图像信息。
- 在导航、描述和意图识别任务中提升性能，最多增10.81%准确率。
- 参数增加不足3%，适合部署在资源受限的机器人系统上。
在共享人类环境中的机器人不仅需导航、交互与感知，还需解读并响应动态且不可预测的人类行为。尽管近期基于视觉语言模型（VLMs）的研究在增强机器人感知与指令理解方面取得进展，但在多模态人机交互（HRI）复杂性面前仍显不足。为此，我们提出一种轻量级的语言到视觉反馈模块，将大语言模型（LLM）与视觉编码器在VLM中形成闭环。该模块通过门控多层感知机（MLP）将图像标记的隐藏状态投影回编码器输入，触发第二轮场景重读，以文本上下文重新解释视觉内容。我们在三个机器人相关任务上评估：模拟环境导航（Habitat）、序列场景描述（Mementos-Robotics）及人类意图识别（自建HRI数据集）。结果表明，该方法使Qwen 2.5（7B）在距离上减少3.3%、描述得分提升0.057、意图识别准确率提高2.93%，额外参数少于3%；Gemma 3（4B）和LLaVA OV 1.5（4B）在导航任务中表现混合，但在后两项任务上分别获得+0.111、+0.055的描述提升及+10.81%、+4.79%的准确率提升。代码已开源。
原文摘要 · Abstract (English)
Robots operating in shared human environments must not only navigate, interact, and detect their surroundings, they must also interpret and respond to dynamic, and often unpredictable, human behaviours. Although recent advances have shown promise in enhancing robotic perception and instruction-following using Vision-Language Models (VLMs), they remain limited in addressing the complexities of multimodal human-robot interactions (HRI). Motivated by this challenge, we introduce a lightweight language-to-vision feedback module that closes the loop between an LLM and the vision encoder in VLMs. The module projects image-token hidden states through a gated Multi-Layer Perceptron (MLP) back into the encoder input, prompting a second pass that reinterprets the scene under text context. We evaluate this approach on three robotics-centred tasks: navigation in a simulated environment (Habitat), sequential scene description (Mementos-Robotics), and human-intention recognition (our HRI dataset). Results show that our method improves Qwen 2.5 (7B) by $3.3\%$ (less distance), $+0.057$ description score, and $+2.93\%$ accuracy, with less than $3\%$ extra parameters; Gemma 3 (4B) and LLaVA OV 1.5 (4B) show mixed navigation results but gains $+0.111,+0.055$ and $+10.81\%,+4.79\%$ on the latter two tasks. Code is available at https://github.com/alessioGalatolo/VLM-Reasoning-for-Robotics
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。