让视觉语言模型的推理更懂图像中物体间关系
ReGuLaR: Relation-Grounded Latent Reasoning for Large Vision-Language Models
- 用关系感知的隐空间推理，让模型关注图像中物体及相互关系
- 在多个基准上超越现有方法，最高提升6.8个百分点
- 适合需要精准视觉推理的应用，如医疗影像分析
链式思维（CoT）通过自然语言描述推理过程，显著提升了大视觉语言模型（LVLMs）的推理能力。然而，这种离散文本推理难以充分编码连续视觉证据。近期工作尝试将推理转移到连续隐空间，但现有方法仍未能有效关联视觉证据中的组合与关系结构。为此，我们提出 ReGuLaR，一种关系基础的隐空间推理框架，显式地将隐状态锚定于关键却常被忽略的视觉关系。ReGuLaR 在训练时使用 ReGFormer 关注与问题相关的物体及其相互关系，推理时无需调用 ReGFormer 即可生成答案。为支持训练，我们构建了 RGROUNDING-351K，一个包含关键物体边界框和物体间关系标注的真实世界视觉语言数据集。在多个基准上的大量实验表明，ReGuLaR 持续优于现有方法，达到当前最优性能。代码将随论文提交，并在录用后公开。
原文摘要 · Abstract (English)
Chain-of-thought (CoT) reasoning has significantly improved the reasoning ability of large vision-language models (LVLMs) by verbalizing intermediate reasoning steps in natural language. However, such discrete textual rationales are often insufficient for encoding continuous visual evidence. Recent work addresses this limitation by moving reasoning into continuous latent space. Despite promising progress, existing methods leave latent reasoning insufficiently connected to the compositional and relational structure of visual evidence. To address this gap, we introduce ReGuLaR, a relation grounded latent reasoning framework that explicitly grounds latent states in these critical yet overlooked visual evidence. ReGuLaR uses a training-time ReGFormer to focus latent reasoning on question-relevant objects and inter-object relations, while at inference time the model reasons and generates answers without invoking the ReGFormer. To support training ReGuLaR, we construct RGROUNDING-351K, a real-world vision-language dataset annotated with key object bounding boxes and inter-object relations. Extensive experiments across diverse benchmarks show that ReGuLaR consistently outperforms existing approaches and achieves state-of-the-art performance. We include our code in the submission and will release the code and training data publicly upon acceptance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。