不训练即可加速视觉语言模型，效率提升显著。
Similarity-Aware Token Pruning: Your VLM but Faster
- 基于相似性与图结构动态调整每层剪枝率，无需额外训练。
- ViT-H/14速度翻倍，图像识别准确率仅降0.6%。
- 适用于各类视觉语言模型，尤其适合部署时追求低延迟的场景。
视觉变换器（ViTs）和视觉语言模型（VLMs）因自注意力机制的二次复杂度面临巨大计算压力。尽管令牌剪枝是可行方案，但现有方法常需额外训练或无法跨层动态适应。本文提出SAINT，一种无需训练的剪枝框架，利用令牌相似性与图结构建模，动态优化剪枝率与冗余阈值。通过系统分析，发现变压器存在统一的三阶段令牌演化过程（对齐-探索-聚合），可在早期阶段进行激进剪枝而不损失关键信息。对于ViT，SAINT在224px分辨率下使ViT-H/14吞吐量翻倍，ImageNet-1K上仅损失0.6%准确率，优于最接近的基线0.8%。对于VLM，SAINT采用三种模式：仅剪ViT、仅剪LLM、混合剪枝。应用于LLaVA-13B时，令牌减少75%，延迟接近LLaVA-7B，跨基准性能损失低于1%。本工作建立了一个统一且实用的高效推理框架，适用于ViTs与VLMs。
原文摘要 · Abstract (English)
The computational demands of Vision Transformers (ViTs) and Vision-Language Models (VLMs) remain a significant challenge due to the quadratic complexity of self-attention. While token pruning offers a promising solution, existing methods often introduce training overhead or fail to adapt dynamically across layers. We present SAINT, a training-free token pruning framework that leverages token similarity and a graph-based formulation to dynamically optimize pruning rates and redundancy thresholds. Through systematic analysis, we identify a universal three-stage token evolution process (aligner-explorer-aggregator) in transformers, enabling aggressive pruning in early stages without sacrificing critical information. For ViTs, SAINT doubles the throughput of ViT-H/14 at 224px with only 0.6% accuracy loss on ImageNet-1K, surpassing the closest competitor by 0.8%. For VLMs, we apply SAINT in three modes: ViT-only, LLM-only, and hybrid. SAINT reduces LLaVA-13B's tokens by 75%, achieving latency comparable to LLaVA-7B with less than 1% performance loss across benchmarks. Our work establishes a unified, practical framework for efficient inference in ViTs and VLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。