用文本语义选关键视觉区域，提升视觉语言模型效率与精度
Semantic-Clipping: Efficient Vision-Language Modeling with Semantic-Guidedd Visual Selection
- 根据文本语义自动筛选图像关键区域，减少冗余视觉令牌
- 在7个基准上平均提升3.3%，细节理解任务提升5.3%
- 无需重训练，可无缝集成到现有视觉语言模型中
视觉语言模型（VLM）通过对齐的视觉编码器将图像转换为视觉标记，使它们能像文本一样被主干大语言模型（LLM）处理，从而在视觉问答（VQA）等任务中表现优异。为提升细粒度视觉推理能力，近期研究引入图像裁剪技术，将所有编码后的子图像输入模型，但该方法显著增加视觉标记数量，导致效率下降并可能干扰LLM。为此，我们提出一种轻量、通用的框架——SEMCLIP，能无缝集成至现有VLM中，通过文本语义识别关键视觉区域，增强模型对细节的理解能力，且无需重训练。该方法还将文本信号融入视觉编码过程，兼顾效率与效果。实验表明，SEMCLIP在7个基准上平均提升LLaVA-1.5（7B）模型性能3.3%，尤其在挑战性细节理解任务V*上提升达5.3%。
原文摘要 · Abstract (English)
Vision-Language Models (VLMs) leverage aligned visual encoders to transform images into visual tokens, allowing them to be processed similarly to text by the backbone large language model (LLM). This unified input paradigm enables VLMs to excel in vision-language tasks such as visual question answering (VQA). To improve fine-grained visual reasoning, recent advancements in vision-language modeling introduce image cropping techniques that feed all encoded sub-images into the model. However, this approach significantly increases the number of visual tokens, leading to inefficiency and potential distractions for the LLM. To address the generalization challenges of image representation in VLMs, we propose a lightweight, universal framework that seamlessly integrates with existing VLMs to enhance their ability to process finegrained details. Our method leverages textual semantics to identify key visual areas, improving VQA performance without requiring any retraining of the VLM. Additionally, it incorporates textual signals into the visual encoding process, enhancing both efficiency and effectiveness. The proposed method, SEMCLIP, strengthens the visual understanding of a 7B VLM, LLaVA-1.5 by 3.3% on average across 7 benchmarks, and particularly by 5.3% on the challenging detailed understanding benchmark V*.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。