让小模型直接学习大模型的特征空间结构，提升无监督压缩效果。
Simple Unsupervised Knowledge Distillation With Space Similarity
- 通过空间相似性损失，让小模型各维度特征对齐大模型。
- 在多个基准上实现优于现有方法的性能，尤其在小模型上优势明显。
- 适合做无标签场景下的模型轻量化，无需人工设计关系规则。
近期研究表明，自监督学习（SSL）难以有效适配小型网络架构。为在无标签条件下训练小型网络，一种可行方案是采用无监督知识蒸馏（UKD）。现有UKD方法通常人为设计教师与学生之间的样本间/样本内关系进行保留，但可能忽略教师映射中其他关键关系。本文不依赖人工构造关系，而是直接促使学生建模教师的嵌入流形。若映射后的流形相似，则所有样本间/样本内关系可间接保留。我们首先证明，先前方法因仅依赖 $L_2$ 归一化的嵌入特征，无法保持教师的潜在流形。随后提出一种简单损失项——空间相似性（space similarity），使学生特征空间的每一维与对应教师维度尽可能相似。大量实验表明，该方法在多个基准上表现优异。
原文摘要 · Abstract (English)
As per recent studies, Self-supervised learning (SSL) does not readily extend to smaller architectures. One direction to mitigate this shortcoming while simultaneously training a smaller network without labels is to adopt unsupervised knowledge distillation (UKD). Existing UKD approaches handcraft preservation worthy inter/intra sample relationships between the teacher and its student. However, this may overlook/ignore other key relationships present in the mapping of a teacher. In this paper, instead of heuristically constructing preservation worthy relationships between samples, we directly motivate the student to model the teacher's embedding manifold. If the mapped manifold is similar, all inter/intra sample relationships are indirectly conserved. We first demonstrate that prior methods cannot preserve teacher's latent manifold due to their sole reliance on $L_2$ normalised embedding features. Subsequently, we propose a simple objective to capture the lost information due to normalisation. Our proposed loss component, termed \textbf{space similarity}, motivates each dimension of a student's feature space to be similar to the corresponding dimension of its teacher. We perform extensive experiments demonstrating strong performance of our proposed approach on various benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。