arXiv:2604.19567cs.AI2026-04

让视觉语言模型学会看图做语义推理,提升机器人理解物体关系的能力。

Multi-modal Reasoning with LLMs for Visual Semantic Arithmetic

论文配图:Multi-modal Reasoning with LLMs for Visual Semantic Arithmetic
图 1 · 摘自论文原文
  • 提出新任务与数据集,系统评测图像语义运算能力。
  • 通过强化学习微调,显著提升模型在图像关系推理上的表现。
  • 适合研究视觉推理、人机交互与服务机器人的学者参考。

强化学习作为后训练方法对提升大语言模型在编程和数学中的推理能力至关重要,但其在视觉语义运算方面的潜力尚未被充分探索。经典文本类比 'king'-'man'+'woman'='queen' 展示了关系推理能力,但若将文本替换为 'king' 和 'man' 的图像,则性能大幅下降,因需常识知识并从冗余视觉细节中提取简洁概念。该能力对非结构化环境中服务与家庭机器人至关重要,例如在厨房中识别 'powder' 与 'cake' 之间存在 'is made of' 关系,可实现工具替换与任务泛化。现有方法在图像特征向量运算后解码,存在模态鸿沟且缺乏系统评估。本文提出两项新任务:两词相减与三词运算,并构建图像关系对数据集(IRPD)用于基准测试。进一步提出语义运算强化微调(SAri-RFT),利用可验证函数与组相对策略优化(GRPO)对大型视觉-语言模型(LVLMs)进行后训练。该方法在IRPD和真实世界数据集Visual7W-Telling上均达到当前最优效果。通过赋予LVLM强大的跨模态关系推理能力,本工作推动家用机器人将符号推理与感知结合,增强复杂环境下的决策、工具适应性与人机交互能力。数据集与源代码见附录。

原文摘要 · Abstract (English)

Reinforcement learning (RL) as post-training is crucial for enhancing the reasoning ability of large language models (LLMs) in coding and math. However, their capacity for visual semantic arithmetic, inferring relationships from images, remains underexplored. The classic text analogy "king"-"man"+"woman" = "queen" illustrates relational reasoning, yet replacing text with images of "king" and "man" significantly reduces performance because it requires commonsense knowledge and the extraction of concise concepts from irrelevant visual details. This capability is important for service and domestic robotics in unstructured environments, where robots must infer semantic relationships among objects, agents, and actions. In a kitchen, recognizing from images that "powder" and "cake" are related by "is made of" grounds symbolic relations in perception, enabling tool substitution, task generalization, and improved semantic reasoning. Prior work approaches semantic arithmetic by decoding image features after vector arithmetic, but suffers from modality gaps and lacks systematic evaluation. In this paper, we formulate two novel tasks, two-term subtraction and three-term operations, and construct the Image-Relation-Pair Dataset (IRPD) for benchmarking. We further propose Semantic Arithmetic Reinforcement Fine-Tuning (SAri-RFT), which post-trains large vision-language models (LVLMs) using a verifiable function and Group Relative Policy Optimization (GRPO). Our method achieves state-of-the-art results on IRPD and the real-world Visual7W-Telling dataset. By equipping LVLMs with robust cross-modal relational reasoning, this work advances domestic robots' ability to ground symbolic reasoning in perception, enhancing decision-making, tool adaptability, and human-robot interaction in complex environments. Datasets and source code are provided in the supplementary material.

视觉推理多模态强化学习机器人

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。