不训练即可剪枝视觉令牌，保留关键前景与背景信息。
What Kind of Visual Tokens Do We Need? Training-free Visual Token Pruning for Multi-modal Large Language Models from the Perspective of Graph
- 基于语义相似性构建视觉令牌图，通过信息传播筛选重要节点。
- 在VQA2.0和TextVQA上减少63.57%计算量，准确率下降不足3%。
- 适合需要高效推理的多模态大模型部署场景。
近期多模态大语言模型（MLLMs）常使用大量视觉令牌以弥补视觉表征不足，导致计算开销过大且存在明显冗余。本文从图视角探究了MLLM所需的视觉令牌类型，发现不同难度样本下前景与背景令牌均至关重要。基于此，提出无需训练的图结构剪枝方法G-Prune：将视觉令牌视为节点，依据语义相似性构建连接，通过加权链接传播信息流，迭代后保留最重要令牌（可为前景或背景）。在最新模型LLaVA-NeXT上验证，该方法在多个基准测试中显著降低计算负担，同时保持高精度。例如，在VQA2.0和TextVQA上分别减少63.57% FLOPs，准确率仅下降0.95%和2.34%。
原文摘要 · Abstract (English)
Recent Multimodal Large Language Models(MLLMs) often use a large number of visual tokens to compensate their visual shortcoming, leading to excessive computation and obvious visual redundancy. In this paper, we investigate what kind of visual tokens are needed for MLLMs, and reveal that both foreground and background tokens are critical for MLLMs given the varying difficulties of examples. Based on this observation, we propose a graph-based method towards training-free visual token pruning, termed G-Prune.In particular, G-Prune regards visual tokens as nodes, and construct their connections based on their semantic similarities. Afterwards, the information flow is propagated via weighted links, and the most important tokens after iterations are kept for MLLMs, which can be front or background.To validate G-Prune, we apply it to a recent MLLM called LLaVA-NeXT, and conduct extensive experiments on a set of benchmarks.The experiment results show that G-Prune can greatly reduce computation overhead while retaining high performance on both coarse- and fine-grained tasks. For instance, G-Prune can reduce 63.57\% FLOPs of LLaVA-NeXT on VQA2.0 and TextVQA with only 0.95\% and 2.34\% accuracy drops, respectively.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。