通过结构重参数化,让ViT模型层数减半仍保持高精度。
Rethinking Vision Transformer Depth via Structural Reparameterization
- 训练时用并行分支替代深层堆叠,推理时合并为单路径结构。
- ViT-Tiny从12层压缩到3层,ImageNet-1K准确率几乎不变。
- 适合移动端部署,推理速度最高提升37%。
Vision Transformer的实际计算开销源于其深层结构,而现有加速方法多聚焦于算法层面优化如令牌剪枝和注意力加速。这留下一个未充分探索的问题:能否在减少堆叠的Transformer层的同时保持相当的表征能力?为此,我们提出一种基于分支的结构重参数化技术,该技术在训练阶段运行。我们的方法利用Transformer块内的并行分支,可系统性地整合为适用于推理部署的精简单路径模型。合并机制通过在非线性组件入口处逐步融合分支,使前馈网络(FFN)和多头自注意力(MHSA)模块均能进行精确数学重参数化,测试时无近似误差。应用于ViT-Tiny时,该框架成功将原始12层架构缩减至6层、4层甚至仅3层,同时维持ImageNet-1K上的分类准确率。压缩后的模型在移动CPU平台上推理速度最高提升37%。研究结果表明,传统认为必须极深的Transformer堆叠可能过于保守,为构建高效视觉Transformer开辟了新路径。
原文摘要 · Abstract (English)
The computational overhead of Vision Transformers in practice stems fundamentally from their deep architectures, yet existing acceleration strategies have primarily targeted algorithmic-level optimizations such as token pruning and attention speedup. This leaves an underexplored research question: can we reduce the number of stacked transformer layers while maintaining comparable representational capacity? To answer this, we propose a branch-based structural reparameterization technique that operates during the training phase. Our approach leverages parallel branches within transformer blocks that can be systematically consolidated into streamlined single-path models suitable for inference deployment. The consolidation mechanism works by gradually merging branches at the entry points of nonlinear components, enabling both feed-forward networks (FFN) and multi-head self-attention (MHSA) modules to undergo exact mathematical reparameterization without inducing approximation errors at test time. When applied to ViT-Tiny, the framework successfully reduces the original 12-layer architecture to 6, 4, or as few as 3 layers while maintaining classification accuracy on ImageNet-1K. The resulting compressed models achieve inference speedups of up to 37% on mobile CPU platforms. Our findings suggest that the conventional wisdom favoring extremely deep transformer stacks may be unnecessarily restrictive, and point toward new opportunities for constructing efficient vision transformers.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。