arXiv:2602.17196cs.CV2026-02被引 1

用矩阵熵找关键视觉标记,高效加速多模态大模型推理。

EntropyPrune: Matrix Entropy Guided Visual Token Pruning for Multimodal Large Language Models

  • 基于矩阵熵识别信息骤降层,确定最佳剪枝时机。
  • 在LLaVA-1.5-7B上减少68.2%计算量,保留96.0%性能。
  • 无需注意力图,通用性强,适用于高分辨率与视频模型。

多模态大语言模型因每张图像需处理数百个视觉标记而带来巨大推理开销。尽管标记剪枝已被证明可加速推理,但何时何地剪枝仍依赖经验性选择。现有方法通常采用静态、人为选定的层数,限制了可解释性与跨模型迁移能力。本文提出一种矩阵熵视角,发现存在一个‘熵坍缩层’(ECL),在此层视觉表征的信息含量出现显著且一致的下降,为剪枝阶段选择提供了原则性依据。基于此,我们提出熵引导剪枝框架EntropyPrune,通过量化单个视觉标记的信息价值,剪除冗余标记,且不依赖注意力图。此外,为实现高效计算,利用双格拉姆矩阵的谱等价性,将熵计算复杂度降低,理论速度提升达64倍。在多个多模态基准上的实验表明,EntropyPrune在准确率与效率上均优于当前最优剪枝方法。在LLaVA-1.5-7B上,实现68.2%的FLOPs减少,同时保持96.0%的原始性能。该方法还能有效推广至高分辨率与视频类模型,展现出强鲁棒性与可扩展性。代码将在https://github.com/YahongWang1/EntropyPrune公开。

原文摘要 · Abstract (English)

Multimodal large language models (MLLMs) incur substantial inference cost due to the processing of hundreds of visual tokens per image. Although token pruning has proven effective for accelerating inference, determining when and where to prune remains largely heuristic. Existing approaches typically rely on static, empirically selected layers, which limit interpretability and transferability across models. In this work, we introduce a matrix-entropy perspective and identify an "Entropy Collapse Layer" (ECL), where the information content of visual representations exhibits a sharp and consistent drop, which provides a principled criterion for selecting the pruning stage. Building on this observation, we propose EntropyPrune, a novel matrix-entropy-guided token pruning framework that quantifies the information value of individual visual tokens and prunes redundant ones without relying on attention maps. Moreover, to enable efficient computation, we exploit the spectral equivalence of dual Gram matrices, reducing the complexity of entropy computation and yielding up to a 64x theoretical speedup. Extensive experiments on diverse multimodal benchmarks demonstrate that EntropyPrune consistently outperforms state-of-the-art pruning methods in both accuracy and efficiency. On LLaVA-1.5-7B, our method achieves a 68.2% reduction in FLOPs while preserving 96.0% of the original performance. Furthermore, EntropyPrune generalizes effectively to high-resolution and video-based models, highlighting the strong robustness and scalability in practical MLLM acceleration. The code will be publicly available at https://github.com/YahongWang1/EntropyPrune.

多模态剪枝推理加速矩阵熵

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。