arXiv:2606.24539cs.CV2026-06

让大模型像人一样看图指物,通过几何推理提升定位精度。

PointVG-R: Internalizing Geometric Reasoning in MLLMs for Precise Pointing Localization via Visual Chain of Thought

论文配图:PointVG-R: Internalizing Geometric Reasoning in MLLMs for Precise Pointing Localization via Visual Chain of Thought
图 1 · 摘自论文原文
  • 设计视觉思维链流程,模拟人类解读指物动作的迭代思考过程。
  • 在mIoU上比基线高15.86分,实现当前最佳性能。
  • 适合需要精准视觉定位的研究者和开发者参考。

基于指物的视觉定位要求模型通过解析视觉场景与指物手势之间的复杂空间关系,精确找到目标物体。传统方法通常将输入图像编码为静态特征表示,并主要在语言域内进行推理,常忽略图像中丰富的感知线索和显式空间几何信息。本文提出一种基于推理引导的多模态大模型PointVG-R,引入几何感知推理机制,使模型能够通过强化学习与冷启动数据的协同策略,以图像为思考媒介进行推理。我们构建了高质量的视觉思维链数据集EgoPoint-CoT,包含详细的推理轨迹,用于监督微调与强化学习。针对训练中学习信号质量差异问题,提出基于组方差的自适应重要性加权策略,动态调整奖励信号以优化学习过程。实验表明,PointVG-R在指标mIoU上相较基线提升15.86分,达到当前最优水平。大量消融实验验证了各模块的有效性。代码已开源。

原文摘要 · Abstract (English)

Pointing-based visual grounding requires models to precisely locate target objects by deciphering complex spatial relationships between the visual scene and pointing gestures. Traditional methods typically encode input images into static feature representations and perform reasoning primarily within the linguistic domain, often overlooking the rich perceptual cues and explicit spatial geometry inherent in images. In this study, we aim to mitigate the cognitive vulnerability of models in interpreting gestural spatial relations by proposing PointVG-R, a reasoning-guided Multi-modal Large Language Model (MLLM). PointVG-R introduces geometric-aware reasoning for pointing-based grounding, enabling the model to think with images through the strategic integration of Reinforcement Learning (RL) and cold-start data. Specifically, we design a novel geometric reasoning pipeline that simulates the iterative cognitive process humans employ when interpreting pointing gestures. Furthermore, we construct EgoPoint-CoT, a high-quality visual Chain-of-Thought (CoT) dataset featuring detailed reasoning trajectories to guide the model via Supervised Fine-Tuning (SFT) and RL. To address the varying quality of learning signals encountered during training, we further propose an Adaptive Importance Weighting strategy based on Group Variance, which dynamically adjusts reward signals to optimize the learning process. Experimental results demonstrate that PointVG-R achieves SOTA performance, outperforming the baseline by $\textbf{15.86}$ points in mIoU. Extensive ablation studies further validate the efficacy of our proposed modules. Code: https://github.com/lingli1724/PointVG-R.

视觉定位几何推理多模态强化学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。