动态剪枝视觉令牌，让多模态推理更高效
VisionPulse: Dynamic Visual Sparsity for Efficient Multimodal Reasoning
- 按解码步骤动态保留关键视觉令牌，而非静态剪枝
- 每步仅保留5%视觉令牌，推理轨迹缩短11.2%且精度不变
- 适合追求推理效率的多模态模型部署场景
随着大模型在多模态任务中快速发展，推理阶段的开销已成为实际部署的关键瓶颈。现有方法通常在预填充阶段剪枝视觉令牌，假设视觉证据在整个推理过程中保持不变。但我们实证发现，视觉证据具有明显的步骤依赖性：每个解码步骤仅有少量视觉令牌关键，且关键集合随推理过程演变。此外，我们识别出一个耦合瓶颈——冗余视觉上下文会引导模型走向与查询无关区域，延长推理路径。基于此洞察，我们提出VisionPulse，一种在推理过程中分步执行视觉令牌剪枝的框架。VisionPulse通过轻量级视觉注意力质量估计每步保留预算，该预算与大型多模态模型（LMMs）的有效视觉令牌使用高度正相关，并在此预算下仅保留最关键的令牌。通过在推理中强制视觉稀疏性，VisionPulse过滤冗余视觉上下文，同时保留相关视觉证据，自然缩短推理轨迹。大量实验表明，VisionPulse每步仅保留5%视觉令牌，推理轨迹缩短11.2%，而精度几乎不变。
原文摘要 · Abstract (English)
With the rapid advancement of large multimodal models (LMMs), inference-time overhead has become a key bottleneck for real-world deployment. Existing methods typically prune visual tokens at prefill, assuming the required visual evidence remains static during reasoning. However, we empirically show that visual evidence is strongly step-dependent: only a sparse subset of visual tokens is critical at each decoding step, and the critical set evolves across reasoning. Furthermore, we identify a coupled bottleneck where redundant visual context can steer the model toward query-irrelevant regions, lengthening the reasoning trace. Guided by these insights, we propose VisionPulse, a step-wise visual token pruning framework during reasoning. VisionPulse computes a lightweight visual attention mass to estimate the step-wise retention budget by exploiting its strong positive correlation with LMMs' effective visual token usage and retain only the most critical tokens under this budget. By enforcing visual sparsity during reasoning, VisionPulse filters redundant visual context while preserving relevant visual evidence, shortening reasoning traces naturally. Extensive experiments show that VisionPulse only retains 5% of visual tokens per step with reasoning traces shortened by 11.2%, while keeping accuracy almost unchanged.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。