通过贡献感知筛选视觉标记,提升大模型推理效率。
CAPA: Contribution-Aware Pruning and FFN Approximation for Efficient Large Vision-Language Models
- 用注意力贡献度替代原始注意力值,更准确识别关键视觉标记
- 在多个基准上实现高效压缩,性能损失小于1.5%且鲁棒性提升
- 适合追求推理加速的视觉语言模型部署场景
大型视觉语言模型的高效推理受限于处理数千个视觉标记的高成本,但哪些标记和计算可安全移除仍不明确。现有方法常以注意力分数评估标记重要性,但其作为实际贡献代理存在偏差。我们发现,将注意力概率与值向量幅值结合的注意力贡献,能更准确衡量视觉标记价值。实证分析显示,视觉注意力区域功能异质:低贡献的“概率堆积区”可安全剪枝,而高贡献的“结构锚点”对模型性能至关重要。此外,我们发现视觉标记关联的前馈网络(FFN)存在显著冗余,尤其在中间层图像标记呈现线性行为。基于此,提出双策略框架CAPA(贡献感知剪枝与FFN近似),在关键功能转换点使用注意力贡献进行视觉标记剪枝,并通过高效线性近似降低FFN计算量。在多种基线与基准测试中,CAPA实现良好效率-性能平衡,且具备更强鲁棒性。
原文摘要 · Abstract (English)
Efficient inference in Large Vision-Language Models is constrained by the high cost of processing thousands of visual tokens, yet it remains unclear which tokens and computations can be safely removed. While attention scores are commonly used to estimate visual token importance, they are an imperfect proxy for actual contribution. We show that Attention Contribution, which weights attention probabilities by value vector magnitude, provides a more accurate criterion for visual token selection. Our empirical analysis reveals that visual attention sinks are functionally heterogeneous, comprising Probability Dumps with low contribution that can be safely pruned, and Structural Anchors with high contribution essential for maintaining model performance. Further, we identify substantial redundancy in Feed-Forward Networks (FFNs) associated with visual tokens, particularly in intermediate layers where image tokens exhibit linear behavior. Based on our findings, we introduce CAPA (Contribution-Aware Pruning and FFN Approximation), a dual-strategy framework that prunes visual tokens using attention contribution at critical functional transitions and reduces FFN computation through efficient linear approximations. Experiments on various benchmarks across baselines show that CAPA achieves competent efficiency--performance trade-offs with improved robustness.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。