通过跨层保留未选视觉标记，提升模型推理效率与细节感知能力
SwiftVLM: Efficient Vision-Language Model Inference via Cross-Layer Token Bypass
- 引入跨层标记旁路机制，避免过早丢弃重要视觉信息
- 在多个模型和数据集上实现更高精度与更低计算开销的平衡
- 无需训练、适配性强，适合需要精细视觉理解的场景
视觉标记剪枝是降低视觉语言模型计算成本的有前景方法，现有方法多依赖早期剪枝决策以提升效率。尽管在粗粒度推理任务中表现良好，但在需细粒度视觉细节的任务上性能显著下降。通过分层分析发现，不同层级的视觉标记重要性差异显著：浅层被认为不重要的标记，在后续文本条件推理中可能变得关键。为避免因过早剪枝导致关键信息不可逆丢失，我们提出一种新剪枝范式——旁路（bypass），保留未选视觉标记并传递至后续剪枝阶段重新评估。基于此，我们设计SwiftVLM，一种简单且无需训练的方法，在具备强视觉标记选择能力的特定层执行剪枝，并支持各层独立决策。在多个视觉语言模型与基准测试上的实验表明，SwiftVLM持续优于现有剪枝策略，实现了更优的精度-效率权衡与更忠实的视觉标记选择行为。
原文摘要 · Abstract (English)
Visual token pruning is a promising approach for reducing the computational cost of vision-language models (VLMs), and existing methods often rely on early pruning decisions to improve efficiency. While effective on coarse-grained reasoning tasks, they suffer from significant performance degradation on tasks requiring fine-grained visual details. Through layer-wise analysis, we reveal substantial discrepancies in visual token importance across layers, showing that tokens deemed unimportant at shallow layers can later become highly relevant for text-conditioned reasoning. To avoid irreversible critical information loss caused by premature pruning, we introduce a new pruning paradigm, termed bypass, which preserves unselected visual tokens and forwards them to subsequent pruning stages for re-evaluation. Building on this paradigm, we propose SwiftVLM, a simple and training-free method that performs pruning at model-specific layers with strong visual token selection capability, while enabling independent pruning decisions across layers. Experiments across multiple VLMs and benchmarks demonstrate that SwiftVLM consistently outperforms existing pruning strategies, achieving superior accuracy-efficiency trade-offs and more faithful visual token selection behavior.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。