通过动态专家激活与路由感知剪枝，显著降低多模态大模型推理开销。
FastMMoE: Accelerating Multimodal Large Language Models through Dynamic Expert Activation and Routing-Aware Token Pruning
- 基于路由分析动态减少视觉令牌的专家计算量。
- 可降低55.0%计算量，保持95.5%原模型性能。
- 无需训练，适配资源受限场景部署。
多模态大语言模型（MLLM）性能优异，但高分辨率视觉输入导致视觉标记序列过长，带来巨大推理延迟。减少冗余视觉标记对缓解计算与内存压力至关重要，有助于在资源受限或低延迟场景中部署。现有方法主要依赖注意力机制进行冗余分析，且针对密集架构。本文提出无需训练的FastMMoE加速框架，从路由分析视角出发，结合两种互补策略：(i) 视觉标记的专家激活削减，以最小化无效专家计算；(ii) 路由感知标记剪枝，利用路由概率分布的相似性识别并移除高度冗余的视觉标记。在DeepSeek-VL2和InternVL3.5等大规模MoE-MLLM上实验表明，FastMMoE可将FLOPs降低最高55.0%，同时保留约95.5%原始性能，优于包括FastV和SparseVLM在内的多个密集模型剪枝基线，在不同保留率下均表现更优。
原文摘要 · Abstract (English)
Multimodal large language models (MLLMs) have achieved impressive performance, but high-resolution visual inputs result in long sequences of visual tokens and substantial inference latency. Reducing redundant visual tokens is critical to ease computational/memory burdens while preserving performance, enabling MLLM deployment in resource-constrained or latency-sensitive scenarios. Current visual token pruning methods mainly rely on attention-based redundancy analysis and are tailored to dense architectures. We propose Fast Multimodal Mixture-of-Experts (FastMMoE), a training-free acceleration framework for mixture-of-experts (MoE) based MLLMs, developed from a routing analysis perspective. FastMMoE combines two complementary strategies: (i) expert activation reduction for visual tokens to minimize unnecessary expert computation; and (ii) routing-aware token pruning that leverages similarity in routing probability distributions to identify and remove highly redundant visual tokens. Experiments on large-scale MoE-MLLMs such as DeepSeek-VL2 and InternVL3.5 demonstrate that FastMMoE can reduce FLOPs by up to 55.0% while retaining approximately 95.5% of the original performance, consistently outperforming dense-model pruning baselines including FastV and SparseVLM across multiple retention rates.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。