通过频域演化分析,高效去除多模态模型冗余视觉令牌。
Spectral Evolution-Guided Token Pruning in Multimodal Large Language Models

- 基于跨层频谱演化判断令牌重要性,避免单层信号偏差。
- 在图像与视频任务上实现高效率压缩,性能不降反升。
- 无需训练,适配多种多模态大模型,加速推理过程。
减少视觉令牌冗余对提升多模态大模型(MLLM)效率至关重要,同时保持跨模态推理性能。现有令牌剪枝方法通常依赖单层信号(如注意力分数或令牌相似度),忽略视觉表示在不同层间的转换特性,且可能引入模态序列中的位置偏差。为此,我们提出一种无需训练的剪枝框架——跨层频谱演化(CLSE)。CLSE不从单层特征幅值衡量令牌重要性,而是量化令牌表示在频率域中随Transformer层数演化的动态变化。该演化反映了从高频结构细节向低频语义抽象的转变过程。我们发现,跨层频谱重分配越强的令牌,越可能具有语义活跃性,应予以保留。通过建模跨层令牌动态,CLSE提供了稳定的重要性评估标准,有效缓解位置偏差。在图像与视频基准上的大量实验表明,CLSE在激进的令牌缩减下仍实现效率与精度的更优权衡。在多个MLLM中,CLSE显著降低计算量(FLOPs)、KV缓存内存占用及延迟,同时保持甚至提升性能。
原文摘要 · Abstract (English)
Reducing visual token redundancy is critical for accelerating Multimodal Large Language Models (MLLMs) without degrading cross-modal reasoning performance. Existing token pruning methods typically rely on single-layer signals, such as attention scores or token similarities, which overlook the cross-layer transformation of visual representations and may exhibit positional bias in multimodal token sequences. To address this limitation, we propose a training-free token pruning framework based on Cross-Layer Spectral Evolution (CLSE). Instead of measuring token importance from single-layer feature magnitudes, CLSE quantifies how token representations evolve across Transformer layers in the frequency domain. This evolution reflects the transition from high-frequency structural details to low-frequency semantic abstractions. We observe that tokens with stronger spectral redistribution across layers are more likely to be semantically active and should therefore be preserved. By modeling cross-layer token dynamics, CLSE provides a stable importance criterion that mitigates positional bias. Extensive experiments on both image and video benchmarks demonstrate that CLSE achieves a superior trade-off between efficiency and accuracy under aggressive token reduction. Across multiple MLLMs, CLSE reduces FLOPs, KV cache memory, and latency while maintaining competitive or improved performance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。