发现视觉变压器压缩中特征蒸馏失效的根本原因并提出轻量修复方案。
From Per-Image Low-Rank to Encoding Mismatch: Rethinking Feature Distillation in Vision Transformers
- 揭示图像级低秩与令牌能量扩散的编码错配现象。
- 提出Lift和WideLast两种轻量修复方法，提升压缩模型准确率。
- 适用于需要高效压缩视觉变压器的场景，如移动端部署。
特征图知识蒸馏在结构相近的视觉变压器（ViTs）间表现良好，但在模型压缩时往往失败。我们重新审视这一现象，发现一个悖论：单张图像经SVD分析高度可压缩，理论上窄学生配合线性投影器应能匹配教师；但数据集层面的PCA分析显示，教师模型由多个低秩子空间构成，且不同输入间存在显著子空间旋转。进一步引入令牌级谱能模式（SEP），发现即使处于低秩子空间，令牌仍会将能量广泛分布于通道模式中，造成带宽错配。我们称此为编码错配。为此提出两种最小化修复方案：(i) Lift在推理时保留轻量升维投影器以拓宽通道，(ii) WideLast仅扩大学生最后一层，实现输入依赖式扩展。在ImageNet-1K上，该方法使从CaiT-S24蒸馏出的DeiT-Tiny准确率从74.86%提升至77.53%/78.23%，且对无蒸馏训练的学生模型也有增益。分析阐明了特征蒸馏失败的条件与修复机制。代码与原始数据见https://github.com/thy960112/From-Per-Image-Low-Rank-to-Encoding-Mismatch。
原文摘要 · Abstract (English)
Feature-map knowledge distillation (KD) transfers internal representations well between comparably sized Vision Transformers (ViTs), but it often fails in compression. We revisit this failure and uncover a paradox. Sample-wise SVD shows that each image is highly compressible, which seems to suggest that a narrow student with a linear projector should match the teacher "in principle". However, a dataset-level view contradicts this intuition: PCA shows that the teacher is a union of low-rank subspaces with significant subspace rotation across inputs. We further introduce token-level Spectral Energy Patterns (SEP) and find an architecture-invariant encoding law: tokens spread energy broadly across channel modes even when they live in low-rank subspace, creating a bandwidth mismatch. We refer to this combined phenomenon as an encoding mismatch. We propose two minimal remedies, Lift or WideLast: (i) Lift retains a lightweight lifting projector at inference to provide wider channel, or (ii) WideLast widens only the student's last block, enabling an input-dependent expansion. On ImageNet-1K, these fixes revive feature KD for ViT compression, improving DeiT-Tiny distilled from CaiT-S24 from 74.86% to 77.53%/78.23% top-1 accuracy, and they also strengthen students trained without distillation. Our analyses clarify when and why feature-map KD fails and then how to fix it. Code and raw data are provided in https://github.com/thy960112/From-Per-Image-Low-Rank-to-Encoding-Mismatch.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。