arXiv:2507.00505cs.CV2025-07ICCV被引 5

通过添加6个空间视觉标记,提升多模态大模型的局部视觉理解能力。

LLaVA-SP: Enhancing Visual Representation with Visual Spatial Tokens for MLLMs

  • 用卷积核从视觉块特征生成空间标记,模拟中心到全局、抽象到具体的视觉顺序。
  • 提出两种变体:裁剪版专注细节,池化版捕捉全局语义,适配不同任务。
  • 仅需微调LoRA,性能超越LLaVA-1.5,推理延迟几乎不变。

多模态大语言模型(MLLMs)通常将基于CLIP-ViT的视觉编码器与大语言模型连接。尽管CLIP-ViT能有效捕捉全局图像特征,但难以建模相邻图像块之间的局部关系,导致视觉表示较弱,影响模型对细节的理解。为此,我们提出LLaVA-SP,仅在原始视觉标记基础上添加六个空间视觉标记以增强视觉表示。该方法具有三大优势:1)提出新型投影器,利用卷积核从ViT块特征中提取空间标记,模拟“从中心区域到全局”和“从抽象到具体”两种视觉空间排序方式,并通过交叉注意力融合细粒度视觉信息,丰富整体视觉表示;2)设计两种模型变体:LLaVA-SP-Cropping通过渐进式裁剪聚焦细节特征,LLaVA-SP-Pooling通过自适应池化捕捉全局语义,使模型能应对多样化的视觉理解任务;3)大量实验表明,使用LoRA微调的LLaVA-SP在多个多模态基准上实现显著性能提升,多项任务优于当前最优的LLaVA-1.5模型,且推理延迟几乎相同。代码与模型已开源于https://github.com/CnFaker/LLaVA-SP。

原文摘要 · Abstract (English)

The architecture of multimodal large language models (MLLMs) commonly connects a vision encoder, often based on CLIP-ViT, to a large language model. While CLIP-ViT works well for capturing global image features, it struggles to model local relationships between adjacent patches, leading to weaker visual representation, which in turn affects the detailed understanding ability of MLLMs. To solve this, we propose LLaVA-SP, which only adds six spatial visual tokens to the original visual tokens to enhance the visual representation. Our approach offers three key advantages: 1) We propose a novel Projector, which uses convolutional kernels to derive visual spatial tokens from ViT patch features, simulating two visual spatial ordering approaches: "from central region to global" and "from abstract to specific". Then, a cross-attention mechanism is applied to fuse fine-grained visual information, enriching the overall visual representation. 2) We present two model variants: LLaVA-SP-Cropping, which focuses on detail features through progressive cropping, and LLaVA-SP-Pooling, which captures global semantics through adaptive pooling, enabling the model to handle diverse visual understanding tasks. 3) Extensive experiments show that LLaVA-SP, fine-tuned with LoRA, achieves significant performance improvements across various multimodal benchmarks, outperforming the state-of-the-art LLaVA-1.5 model in multiple tasks with nearly identical inference latency. The code and models are available at https://github.com/CnFaker/LLaVA-SP.

多模态视觉编码空间建模模型优化

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。