通过分组并行调度视觉标记，大幅加速多模态模型推理。
Parallel Vision Token Scheduling for Fast and Accurate Multimodal LMMs Inference
- 将视觉标记分为主体与非主体组，分路并行处理
- 最多剪掉88.9%视觉标记，仅轻微损失精度
- 无需训练、适配多种模型架构，适合部署优化
多模态大语言模型在视觉-语言推理中表现优异，但因自注意力机制随序列长度呈平方增长，且高分辨率图像带来数千个视觉标记，导致推理延迟严重。直接剪枝低信息量标记虽可缓解负担，但盲目移除可能削弱背景或细粒度问题所需的上下文线索，影响准确性。本文提出ParVTS（Parallel Vision Token Scheduling），一种无需训练的调度框架：将视觉标记划分为主体与非主体组，分路并行处理，将语义传递至问题标记，并在推理中途丢弃非主体路径以减少计算量。该方法降低计算复杂度，无需启发式规则或额外模块，兼容多种现有多模态模型架构。实验表明，ParVTS在多个多模态模型骨干网络上可剪枝高达88.9%的视觉标记，性能下降极小，实现1.77倍加速和70%浮点运算量（FLOPs）减少。
原文摘要 · Abstract (English)
Multimodal large language models (MLLMs) deliver impressive vision-language reasoning but suffer steep inference latency because self-attention scales quadratically with sequence length and thousands of visual tokens contributed by high-resolution images. Naively pruning less-informative visual tokens reduces this burden, yet indiscriminate removal can strip away contextual cues essential for background or fine-grained questions, undermining accuracy. In this paper, we present ParVTS (Parallel Vision Token Scheduling), a training-free scheduling framework that partitions visual tokens into subject and non-subject groups, processes them in parallel to transfer their semantics into question tokens, and discards the non-subject path mid-inference to reduce computation. This scheduling reduces computational complexity, requires no heuristics or additional modules, and is compatible with diverse existing MLLM architectures. Experiments across multiple MLLM backbones show that ParVTS prunes up to 88.9% of visual tokens with minimal performance drop, achieving 1.77x speedup and 70% FLOPs reduction.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。