通过预投影筛选关键视觉标记，显著提升多模态大模型效率与精度。
ReDiPrune: Relevance-Diversity Pre-Projection Token Pruning for Efficient Multimodal LLMs
- 在视觉编码器输出阶段直接筛选信息量高的标记，不依赖模型重训练。
- 仅保留15%视觉标记时，计算量减少6倍以上，准确率反而提升2.0%。
- 无需修改模型结构，可即插即用，适合视频和图像多模态任务部署。
当前多模态大模型因需处理大量视觉标记而计算成本高。本文提出ReDiPrune，一种应用于视觉-语言投影器之前的训练无关标记剪枝方法，保持视觉特征丰富且具有判别性。不同于对压缩表示进行剪枝的后置方法，ReDiPrune直接从视觉编码器输出中选择信息丰富的标记，保留精细的空间与语义线索。每个标记由轻量规则评分，综合考虑文本条件下的相关性与最大最小多样性，确保所选标记既相关又无冗余。该方法完全即插即用，无需重新训练或架构修改，可无缝插入编码器与投影器之间。在四个视频和五个图像基准上均一致改善了准确率-效率权衡。例如，在EgoSchema数据集上使用LLaVA-NeXT-Video-7B模型，仅保留15%视觉标记即可实现+2.0%绝对准确率提升，同时计算量降低超过6倍（以TFLOPs计）。代码已开源：https://github.com/UA-CVML/ReDiPrune。
原文摘要 · Abstract (English)
Recent multimodal large language models are computationally expensive because Transformers must process a large number of visual tokens. We present ReDiPrune, a training-free token pruning method applied before the vision-language projector, where visual features remain rich and discriminative. Unlike post-projection pruning methods that operate on compressed representations, ReDiPrune selects informative tokens directly from vision encoder outputs, preserving fine-grained spatial and semantic cues. Each token is scored by a lightweight rule that jointly consider text-conditioned relevance and max-min diversity, ensuring the selected tokens are both query-relevant and non-redundant. ReDiPrune is fully plug-and-play, requiring no retraining or architectural modifications, and can be seamlessly inserted between the encoder and projector. Across four video and five image benchmarks, it consistently improves the accuracy-efficiency trade-off. For example, on EgoSchema with LLaVA-NeXT-Video-7B, retaining only 15% of visual tokens yields a +2.0% absolute accuracy gain while reducing computation by more than $6\times$ in TFLOPs. Code is available at https://github.com/UA-CVML/ReDiPrune.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。