根据特征演化层次,动态调整视觉变压器的冗余剪枝与合并策略。
RAPID: Layer-Wise Redundancy-Aware Pruning and Importance-Driven Token Merging for Efficient ViT

- 分层设计剪枝与合并:浅层用冗余-相似度剪枝局部冗余,深层用重要性-相似度合并保护关键语义
- 在极端压缩下比ToMe高4.29%准确率,图像分类任务中实现更优压缩-精度权衡
- 无需训练,可直接部署于ViT和DeiT,适合追求高效推理的模型优化场景
视觉变压器(ViTs)性能强大但因自注意力复杂度为二次方导致计算开销高。虽有剪枝与合并等令牌缩减技术缓解此问题,但通常忽略表征随网络深度演化的特性。本文提出RAPID,一种面向深度的令牌缩减框架,依据各层令牌表征特征自适应调整缩减策略。浅至中层采用冗余-相似度感知剪枝指标,剔除重复的局部模式;深层则转为重要性-相似度感知合并机制,利用分类(CLS)令牌注意力权重保护语义关键令牌,融合不重要且相似的邻居。在ImageNet-1K上对ViT与DeiT架构的实证验证表明,相比即插即用基线如ToMe和ToFu,RAPID在精度-压缩权衡上表现更优,尤其在激进压缩条件下,最高较ToMe提升4.29%准确率。该框架提供了一种无需训练的模型优化模板,使缩减策略与层级特征演化对齐。
原文摘要 · Abstract (English)
Vision Transformers (ViTs) achieve strong performance but suffer from high computational costs due to quadratic self-attention complexity. Although token reduction techniques such as pruning and merging mitigate this, they typically overlook how representations evolve across network depth. We propose RAPID, a depth-aware token reduction framework that adapts reduction strategies to the layer-wise characteristics of token representations. The primary methodological contribution is a bifurcated strategy: in shallow-to-middle layers, RAPID employs a redundancy-similarity aware pruning metric to eliminate over-represented local patterns. As features transition to global semantic concepts in deeper layers, the framework shifts to an importance-similarity aware merging mechanism. This stage leverages classification (CLS) token attention weights to protect semantically critical tokens while fusing less important but similar neighbors. Empirical validation on ImageNet-1K using ViT and DeiT architectures demonstrates that RAPID establishes a superior accuracy-compression Pareto frontier compared to plug-and-play baselines such as ToMe and ToFu. RAPID is particularly robust in aggressive compression regimes, achieving up to 4.29% higher accuracy than ToMe at extreme reduction rates. Our framework provides a training-free template for optimizing vision models by aligning reduction strategies with hierarchical feature evolution.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。