通过精简视觉令牌提升视觉-专家关联性,显著优化多模态模型部署效率。
VisMMOE: Exploiting Visual-Expert Affinity for Efficient Visual-Language MoE Offloading

- 基于视觉-专家亲和性,动态压缩冗余视觉令牌以稳定专家调用模式。
- 在有限内存下实现最高2.68倍的推理加速,且保持高精度。
- 适合资源受限场景下的视觉-语言大模型高效部署,如边缘设备。
大规模视觉-语言混合专家(VL-MoE)模型具备强大的多模态能力,但在内存受限平台上的高效部署仍具挑战。现有MoE卸载系统主要针对文本密集型任务设计,在视觉输入占比高时效果大幅下降,因大量视觉令牌导致专家访问范围广且不可预测。本文提出VisMMoE,其核心思想是:通过剪枝冗余视觉令牌,不仅减少计算量,还能重塑专家需求分布。我们称之为‘视觉-专家亲和性’——令牌剪枝使专家访问更集中于层内、跨层更稳定,从而形成更小且可预测的专家工作集。基于此,VisMMoE结合亲和性感知的令牌压缩、前瞻专家预测及缓存/流水线协同调度,增强专家局部性与预取有效性。我们在多个框架上实现并评估了该系统,结果表明,在代表性VL-MoE模型与基准测试中,相比强基线,端到端推理性能提升最高达2.68倍和1.61倍,同时保持优异准确率。
原文摘要 · Abstract (English)
Large-scale vision-language mixture-of-experts (VL-MoE) models provide strong multimodal capability, but efficient deployment on memory-constrained platforms remains difficult. Existing MoE offloading systems are largely designed for text-centric workloads and become much less effective for visual-heavy inputs, where large numbers of visual tokens induce broader and less predictable expert accesses. We present VisMMoE, a VL-MoE offloading system built on a single systems insight: pruning redundant visual tokens can improve offloading not only by reducing computation, but also by reshaping expert demand. We refer to this effect as \textit{visual-expert affinity}: token pruning makes expert accesses more concentrated within layers and more stable across layers, producing a smaller and more predictable expert working set. Guided by this insight, VisMMoE combines affinity-aware token compression, lookahead expert prediction, and cache/pipeline orchestration to improve expert locality and prefetch effectiveness under tight memory budgets. We implement VisMMoE on multiple frameworks and evaluate it on representative VL-MoE models and benchmarks. VisMMoE improves end-to-end inference performance by up to 2.68x and 1.61x, respectively, over strong baselines for today's VL-MoE deployments while maintaining competitive accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。