arXiv:2503.15426cs.CVcs.AI2025-03被引 19

给大模型加空间坐标提示,让图文定位更准。

Visual Position Prompt for MLLM based Visual Grounding

  • 用可学习的视觉位置提示,为图像注入结构化空间信息
  • 在0.6M数据上训练,性能超越现有模型且支持零样本泛化
  • 适合需要精准图文对齐的视觉理解任务

尽管多模态大语言模型在图像相关任务中表现优异,但在需定位的任务(如视觉定位)中仍难以精确对齐坐标与图像空间信息。这源于两点:一是缺乏显式空间参考,二是特征提取侧重全局上下文而忽略细粒度空间细节。为此,我们提出VPP-LLaVA,通过引入视觉位置提示(VPP)增强模型定位能力。VPP包含全局提示(叠加可学习轴向张量)和局部提示(位置感知查询),分别提供结构化空间线索与精细定位支持。为高效训练,我们构建了包含0.6M高质量样本的VPP-SFT数据集,规模远小于其他模型(如MiniGPT-v2的~21M),但仍显著提升性能。VPP-LLaVA在标准基准上达到顶尖水平,并展现出强大的零样本泛化能力。代码与数据集已开源。

原文摘要 · Abstract (English)

Although Multimodal Large Language Models (MLLMs) excel at various image-related tasks, they encounter challenges in precisely aligning coordinates with spatial information within images, particularly in position-aware tasks such as visual grounding. This limitation arises from two key factors. First, MLLMs lack explicit spatial references, making it difficult to associate textual descriptions with precise image locations. Second, their feature extraction processes prioritize global context over fine-grained spatial details, leading to weak localization capability. To address these issues, we introduce VPP-LLaVA, an MLLM enhanced with Visual Position Prompt (VPP) to improve its grounding capability. VPP-LLaVA integrates two complementary mechanisms: the global VPP overlays a learnable, axis-like tensor onto the input image to provide structured spatial cues, while the local VPP incorporates position-aware queries to support fine-grained localization.To effectively train our model with spatial guidance, we further introduce VPP-SFT, a curated dataset of 0.6M high-quality visual grounding samples. Designed in a compact format, it enables efficient training and is significantly smaller than datasets used by other MLLMs (e.g., ~21M samples in MiniGPT-v2), yet still provides a strong performance boost. The resulting model, VPP-LLaVA, not only achieves state-of-the-art results on standard visual grounding benchmarks but also demonstrates strong zero-shot generalization to challenging unseen datasets. The code and dataset are available at https://github.com/WayneTomas/VPP-LLaVA.

视觉定位多模态空间感知提示学习

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。