arXiv:2603.28069cs.CVcs.AI2026-03被引 5

用视觉标记直接定位目标,比生成坐标更高效准确

MolmoPoint: Better Pointing for VLMs with Grounding Tokens

  • 用特殊标记直接选择图像中的视觉片段进行定位
  • 图像定位达70.7%(PointBench),GUI定位61.1%(ScreenSpotPro)
  • 适合追求高精度定位与低计算开销的视觉语言模型应用

视觉-语言模型(VLMs)的定位能力日益重要。现有方法通过生成坐标作为文本输出的一部分进行定位,需学习复杂坐标系且消耗大量标记。本文提出一种更直观的定位机制:直接选择包含目标概念的视觉标记。模型生成特殊指向标记,跨注意力机制作用于输入图像或视频标记,并选出对应区域。为提升细粒度,后续添加两个特殊标记:第一个选定区域内的子块,第二个指定子块内的具体位置。实验表明,按一致顺序逐点生成、编码前一位置相对关系,并引入“无更多点”类别可进一步提升性能。该方法在图像定位任务上达到70.7%(PointBench),全开源模型中在GUI定位任务上达61.1%(ScreenSpotPro),视频定位人类偏好胜率59.1%(优于文本坐标基线),视频追踪提升6.3%(Molmo2Track)。同时实现更高样本效率,且带来定性差异。

原文摘要 · Abstract (English)

Grounding has become a fundamental capability of vision-language models (VLMs). Most existing VLMs point by generating coordinates as part of their text output, which requires learning a complicated coordinate system and results in a high token count. Instead, we propose a more intuitive pointing mechanism that directly selects the visual tokens that contain the target concept. Our model generates a special pointing token that cross-attends to the input image or video tokens and selects the appropriate one. To make this model more fine-grained, we follow these pointing tokens with an additional special token that selects a fine-grained subpatch within the initially selected region, and then a third token that specifies a location within that subpatch. We further show that performance improves by generating points sequentially in a consistent order, encoding the relative position of the previously selected point, and including a special no-more-points class when selecting visual tokens. Using this method, we set a new state-of-the-art on image pointing (70.7% on PointBench), set a new state-of-the-art among fully open models on GUI pointing (61.1% on ScreenSpotPro), and improve video pointing (59.1% human preference win rate vs. a text coordinate baseline) and tracking (+6.3% gain on Molmo2Track). We additionally show that our method achieves much higher sample efficiency and discuss the qualitative differences that emerge from this design change.

视觉定位视觉语言模型标记选择高效推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。