通过排序筛选关键图像块，让CLIP模型更省计算量
Patch Ranking: Efficient CLIP by Learning to Rank Local Patches
- 用贪心搜索找出最优图像块顺序，构建'黄金排序'
- 剪掉40%图像块后，7个数据集平均精度仅降0.3%
- 加入可学习视觉块，补偿剪枝损失并可能提升性能
对比图像-文本预训练模型如CLIP在下游任务中表现出色，但其视觉变压器（ViT）主干网络计算开销大。现有提升效率的方法多聚焦于剪枝图像块，却未能充分考虑CLIP的多模态特性，也难以确定最优块子集。为此，我们提出贪心搜索方法建立“黄金排序”，并设计轻量级预测器近似该排序。为弥补剪枝带来的性能下降，引入可学习视觉块以恢复甚至增强模型表现。本研究系统性地探索了在CLIP模型的ViT主干中剪枝图像块的可行性。通过框架实现，在保留40%图像块的情况下，7个数据集上平均准确率仅下降0.3%。该工作为构建高效且高性能的多模态模型奠定了基础，解决了先进视觉-语言模型应用中的关键计算瓶颈。
原文摘要 · Abstract (English)
Contrastive image-text pre-trained models such as CLIP have shown remarkable adaptability to downstream tasks. However, they face challenges due to the high computational requirements of the Vision Transformer (ViT) backbone. Current strategies to boost ViT efficiency focus on pruning patch tokens but fall short in addressing the multimodal nature of CLIP and identifying the optimal subset of tokens for maximum performance. To address this, we propose greedy search methods to establish a "Golden Ranking" and introduce a lightweight predictor specifically trained to approximate this Ranking. To compensate for any performance degradation resulting from token pruning, we incorporate learnable visual tokens that aid in restoring and potentially enhancing the model's performance. Our work presents a comprehensive and systematic investigation of pruning tokens within the ViT backbone of CLIP models. Through our framework, we successfully reduced 40% of patch tokens in CLIP's ViT while only suffering a minimal average accuracy loss of 0.3 across seven datasets. Our study lays the groundwork for building more computationally efficient multimodal models without sacrificing their performance, addressing a key challenge in the application of advanced vision-language models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。