用SVD筛选关键视觉令牌，大幅降低模型计算量
Beyond Attention Scores: SVD-Based Vision Token Pruning for Efficient Vision-Language Models
- 基于奇异值分解提取全局重要性，避免局部偏差
- 在仅保留16或32个视觉令牌时仍保持强性能
- 无需训练、可直接插入现有模型，适合资源受限场景
视觉语言模型（VLMs）通过联合处理视觉与文本信息推动了多模态学习的发展，但其对长序列视觉令牌的高计算与内存需求带来了挑战。现有方法依赖注意力分数或令牌范数等局部启发式准则，存在位置偏差和信息分散问题，难以在高剪枝率下保留关键内容，导致在视觉细节丰富的图像上性能下降。为此，我们提出SVD-Prune，一种无需训练、可即插即用的视觉令牌剪枝方法。该方法对视觉令牌特征矩阵进行奇异值分解，利用统计杠杆得分选取前k个最重要的令牌，确保仅保留对主导全局方差贡献最大的内容。实验表明，在极端视觉令牌预算下，SVD-Prune始终优于现有剪枝方法，在仅保留32个或16个视觉令牌时仍能维持良好性能。
原文摘要 · Abstract (English)
Vision-Language Models (VLMs) have revolutionized multi-modal learning by jointly processing visual and textual information. Yet, they face significant challenges due to the high computational and memory demands of processing long sequences of vision tokens. Many existing methods rely on local heuristics, such as attention scores or token norms. However, these criteria suffer from positional bias and information dispersion, limiting their ability to preserve essential content at high pruning ratios and leading to performance degradation on visually detailed images. To address these issues, we propose SVD-Prune, a training-free, plug-and-play token pruning method based on Singular Value Decomposition. It decomposes the vision token feature matrix and selects the top-k tokens using statistical leverage scores, ensuring only tokens contributing most to the dominant global variance are preserved. Experiments show that SVD-Prune consistently outperforms prior pruning methods under extreme vision token budgets, maintaining strong performance even with 32 and 16 vision tokens.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。