减少视觉标记不总能提速，关键看决策位置和硬件。
When Do Fewer Visual Tokens Accelerate Multimodal Inference? A Break-Even Study Across Decision Locations and Hardware
- 按阶段分解延迟，分析决策开销与共享计算
- 预视觉路由在高端显卡上显著低于全量输入延迟
- 适合优化多模态推理的工程师和硬件研究者
减少视觉标记并不一定降低端到端延迟。本文通过可复现的评估协议，考虑决策开销、共享计算及各策略可跳过的算子，对延迟进行阶段级分解。在30个样本的初步实验中，两种自回归探测器仍比全量输入慢，即使重用状态。一个轻量级后视觉预测器在RTX 3090和A100上均得到置信区间低于零的结果，经保守的多重比较校正后仍显著。一个预视觉图像尺寸规则在两块GPU上也得到低于零的区间，但校正后不再显著。预视觉路由相比后视觉剪枝有结构优势：可跳过预处理和视觉编码。在A100上，此优势超过后视觉策略带来的近八倍下游标记减少。报告性能基于全量模型正确回答的样本，非整体基准准确率。
原文摘要 · Abstract (English)
Fewer visual tokens do not guarantee lower end-to-end latency. We evaluate break-even with a reproducible protocol that accounts for decision overhead, shared work, and the operators each policy can avoid. A stage-level decomposition reconciles these components with measured end-to-end latency. In a 30-example pilot, the two tested autoregressive probes remain slower than Full despite state reuse. A lightweight post-vision predictor yields paired confidence intervals below zero on RTX 3090 and A100 and remains significant after a conservative all-pairs Holm correction. A pre-vision image-size rule also yields intervals below zero on both GPUs, although neither comparison remains significant after the same correction. Pre-vision routing has a structural opportunity unavailable to post-vision pruning: it can avoid preprocessing and vision encoding. On A100, this opportunity outweighs a nearly eightfold larger downstream token reduction by the post-vision policy. Reported quality is conditional on examples answered correctly by Full and is not benchmark accuracy.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。