用问题引导压缩视觉令牌，8倍减少计算量且不损失答题精度
QG-VTC: Question-Guided Visual Token Compression in MLLMs for Efficient VQA
- 根据问题与视觉令牌的相关性，动态筛选关键信息
- 仅用1/8视觉令牌即可达到未压缩模型的准确率
- 适合需要高效视觉问答的部署场景
多模态大模型在开放世界视觉问答任务中取得显著进展，但视觉信息的引入导致处理令牌数量激增，带来更高的显存占用和计算开销。图像通常包含比文本更多的冗余信息，且并非所有视觉细节都对特定问题相关。为此，我们提出QG-VTC，一种面向多模态大模型视觉问答任务的问题引导式视觉令牌压缩方法。该方法利用预训练文本编码器与可学习前馈层，将用户问题嵌入视觉编码器的特征空间，并计算问题嵌入与视觉令牌间的相关性得分。通过选择最相关令牌并软性压缩其余部分，QG-VTC确保对用户需求的高度聚焦。此外，采用渐进策略在不同视觉编码层逐步压缩令牌数，最大化保留问题相关信息的同时丢弃无关细节。实验结果表明，本方法仅需1/8的视觉令牌即可实现与未压缩模型相当的性能。代码与模型将公开于GitHub。
原文摘要 · Abstract (English)
Recent advances in Multi-modal Large Language Models (MLLMs) have shown significant progress in open-world Visual Question Answering (VQA). However, integrating visual information increases the number of processed tokens, leading to higher GPU memory usage and computational overhead. Images often contain more redundant information than text, and not all visual details are pertinent to specific questions. To address these challenges, we propose QG-VTC, a novel question-guided visual token compression method for MLLM-based VQA tasks. QG-VTC employs a pretrained text encoder and a learnable feed-forward layer to embed user questions into the vision encoder's feature space then computes correlation scores between the question embeddings and visual tokens. By selecting the most relevant tokens and softly compressing others, QG-VTC ensures fine-tuned relevance to user needs. Additionally, a progressive strategy applies this compression across different vision encoder layers, gradually reducing token numbers. This approach maximizes retention of question-relevant information while discarding irrelevant details. Experimental results show that our method achieves performance on par with uncompressed models using just 1/8 of the visual tokens. The code and model will be publicly available on GitHub.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。