通过预测多层隐藏表示，让模型同时学习多层次抽象特征。
Self-Distillation of Hidden Layers for Self-Supervised Representation Learning
- 用教师模型多层隐藏表示作为目标，引导学生模型学习
- 在ImageNet等数据集上比I-JEPA高10%分类准确率
- 适合追求高精度自监督表征的视觉任务研究者
当前自监督学习（SSL）主要分为生成式方法（如MAE）和预测式方法（如I-JEPA）。生成式方法基于真实数据重建，训练稳定但计算效率低，且不关注高层概念特征；预测式方法依赖最终层自蒸馏，常因目标非平稳而训练不稳定。本文提出Bootleg，让模型预测教师模型多个隐藏层的潜在表示，通过分层目标迫使模型同时捕捉不同抽象层次的特征。实验表明，Bootleg在冻结探针分类任务中显著优于基线，尤其在ImageNet-1K、iNaturalist-21和VTAB上比I-JEPA高出10%，在ADE20K、Cityscapes和COCO-Stuff语义分割任务上也表现优异。
原文摘要 · Abstract (English)
The landscape of self-supervised learning (SSL) is currently dominated by generative approaches (e.g. MAE) that reconstruct raw low-level data, and predictive approaches (e.g. I-JEPA) that predict high-level abstract embeddings. While generative methods are stable due to their reliable training targets based on ground-truth data, they are computationally inefficient for high-redundancy modalities like imagery, and their training objective does not prioritize learning high-level, conceptual features. Conversely, predictive methods often suffer from training instability due to their reliance on the non-stationary targets of final-layer self-distillation. We introduce Bootleg, a method that bridges this divide by tasking the model with predicting latent representations from multiple hidden layers of a teacher. This hierarchical objective forces the model to capture features at varying levels of abstraction simultaneously. We demonstrate Bootleg significantly outperforms comparable baselines (+10% vs. I-JEPA) on frozen probe classification of ImageNet-1K, iNaturalist-21, and VTAB, and semantic segmentation of ADE20K, Cityscapes, and COCO-Stuff.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。