将ViT的注意力知识蒸馏到线性速度的模型，提升高分辨率图像推理效率。
ViT-Linearizer: Distilling Quadratic Knowledge into Linear-Time Vision Models
- 用激活匹配与掩码预测，把ViT的二次复杂度知识转给轻量学生模型
- 在高分辨率任务上实现显著加速，推理速度更快且保持高精度
- 适合追求高效视觉模型的工程师和研究者，尤其关注硬件部署
视觉变换器（ViTs）通过全局自注意力机制取得显著进展，但其二次复杂度在高分辨率输入下可能成为瓶颈。本文提出ViT-Linearizer，一种跨架构知识蒸馏框架，将丰富维特表示迁移到线性时间、循环式结构的学生模型中。方法结合1）激活匹配，即中间层约束使学生对齐教师的逐标记依赖关系；2）掩码预测，要求学生重建教师对未见（掩码）标记的表示，从而有效将二次自注意力知识注入学生模型，同时保持高效计算复杂度。实验表明，该方法在高分辨率任务中带来显著加速，有效缓解推理阶段的硬件压力。此外，该方法还显著提升基于Mamba的架构在标准视觉基准上的表现，在基础尺寸模型上于ImageNet达到84.3%的顶级准确率。结果表明，基于RNN的解决方案在大规模视觉任务中具有巨大潜力，弥合理论效率与实际应用之间的差距。
原文摘要 · Abstract (English)
Vision Transformers (ViTs) have delivered remarkable progress through global self-attention, yet their quadratic complexity can become prohibitive for high-resolution inputs. In this work, we present ViT-Linearizer, a cross-architecture distillation framework that transfers rich ViT representations into a linear-time, recurrent-style model. Our approach leverages 1) activation matching, an intermediate constraint that encourages student to align its token-wise dependencies with those produced by the teacher, and 2) masked prediction, a contextual reconstruction objective that requires the student to predict the teacher's representations for unseen (masked) tokens, to effectively distill the quadratic self-attention knowledge into the student while maintaining efficient complexity. Empirically, our method provides notable speedups particularly for high-resolution tasks, significantly addressing the hardware challenges in inference. Additionally, it also elevates Mamba-based architectures' performance on standard vision benchmarks, achieving a competitive 84.3% top-1 accuracy on ImageNet with a base-sized model. Our results underscore the good potential of RNN-based solutions for large-scale visual tasks, bridging the gap between theoretical efficiency and real-world practice.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。