提出新方法提升多模态大模型推理效率，适配不同架构且性能损失小。
From Inheritance to Saturation: Disentangling the Evolution of Visual Redundancy for Architecture-Aware MLLM Inference Acceleration
- 分解视觉冗余为通用与架构相关两部分，分阶段处理。
- 在Qwen25-VL上实现4.1倍加速，性能保留96.8%。
- 适用于多种模型架构，尤其对Qwen类模型效果显著。
高分辨率多模态大模型因视觉标记数量激增，在推理时面临高昂计算成本。现有加速策略如标记剪枝或层稀疏性存在严重“主干依赖”问题，仅在Vicuna或Mistral等架构（如LLaVA）上表现良好，迁移到Qwen等架构时性能大幅下降。为此，我们利用截断矩阵熵揭示了通用的三阶段推理生命周期，将视觉冗余解耦为通用的内在视觉冗余（IVR）和架构相关的二次饱和冗余（SSR）。基于此，提出HalfV框架：先通过统一剪枝策略缓解IVR，再根据具体表现自适应处理SSR。实验表明，HalfV在多种主干网络上均取得更优的效率-性能权衡。尤其在Qwen25-VL上，实现4.1倍FLOPs加速，性能保留96.8%，显著优于现有最先进方法。代码已开源。
原文摘要 · Abstract (English)
High-resolution Multimodal Large Language Models (MLLMs) face prohibitive computational costs during inference due to the explosion of visual tokens. Existing acceleration strategies, such as token pruning or layer sparsity, suffer from severe "backbone dependency", performing well on Vicuna or Mistral architectures (e.g., LLaVA) but causing significant performance degradation when transferred to architectures like Qwen. To address this, we leverage truncated matrix entropy to uncover a universal three-stage inference lifecycle, decoupling visual redundancy into universal Intrinsic Visual Redundancy (IVR) and architecture-dependent Secondary Saturation Redundancy (SSR). Guided by this insight, we propose HalfV, a framework that first mitigates IVR via a unified pruning strategy and then adaptively handles SSR based on its specific manifestation. Experiments demonstrate that HalfV achieves superior efficiency-performance trade-offs across diverse backbones. Notably, on Qwen25-VL, it retains 96.8\% performance at a 4.1$\times$ FLOPs speedup, significantly outperforming state-of-the-art baselines. Our code is available at https://github.com/civilizwa/HalfV.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。