arXiv:2606.31982cs.CV2026-06

提出新方法在不训练情况下高效压缩视觉令牌,避免注意力失真。

ERA: Entropy-Guided Visual Token Pruning with Rectified Attention for Efficient MLLMs

论文配图:ERA: Entropy-Guided Visual Token Pruning with Rectified Attention for Efficient MLLMs
图 1 · 摘自论文原文
  • 用熵引导选择关键视觉令牌,兼顾多样性与重要性。
  • 通过修复注意力对齐偏差,在压缩后仍保持准确识别能力。
  • 适用于图像、多图和视频,适合追求推理加速的模型部署者。

多模态大语言模型因长视觉令牌序列导致高昂推理成本。无训练的视觉令牌压缩可提供高效解决方案,但现有方法会扭曲注意力分布,引发我们称之为注意力逻辑坍缩的现象。为此,本文提出ERA——一种基于熵引导的视觉令牌剪枝框架,结合修正注意力机制以实现高效多模态大模型。ERA包含三个核心组件:双视角熵剪枝(DEP)、偏置感知令牌回收(BTR)与逻辑保持注意力修正(LAR)。首先,DEP通过联合建模视觉多样性和头级显著性识别代表性锚点令牌;随后,BTR将被剪枝的令牌回收至对应锚点,并估计聚类级别的逻辑偏置;在此基础上,LAR将估计的偏置注入注意力逻辑,有效修正因令牌压缩导致的坍缩问题。三者协同,即使在激进压缩下也能保留视觉证据,在多种多模态大模型上实现单图、多图及视频任务的鲁棒性能。除了实际加速效果,ERA还确立了逻辑保持型视觉令牌剪枝的理论基础与实用范式,统一了算法设计与部署实践。代码已开源:https://github.com/924973292/ERA。

原文摘要 · Abstract (English)

Multimodal Large Language Models (MLLMs) incur prohibitive inference costs due to long visual token sequences. Training-free visual token reduction provides an efficient solution. However, existing methods distort attention distributions, giving rise to a phenomenon we term Attention Logit Collapse. To address this issue, we propose ERA, an Entropy-guided visual token pruning framework with Rectified Attention for efficient MLLMs. Specifically, ERA comprises three crucial components: Dual-view Entropy Pruning (DEP), Bias-aware Token Recycling (BTR), and Logit-preserving Attention Rectification (LAR). First, DEP identifies representative anchor tokens by jointly modeling visual diversity and head-wise saliency. BTR then recycles pruned tokens into their corresponding anchors while estimating a cluster-level logit bias. Building upon this, LAR injects the estimated bias into attention logits, effectively rectifying the collapse induced by token reduction. Together, these components preserve visual evidence even under aggressive compression, enabling robust performance across single-image, multi-image, and video settings on a wide range of MLLMs. Beyond delivering practical acceleration, ERA establishes logit-preserving visual token pruning as a principled framework for efficient MLLMs, unifying theoretical foundation, algorithmic design, and practical deployment. The code is at https://github.com/924973292/ERA.

视觉剪枝多模态推理加速

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。