用少量标注+大量无标注数据，把大模型压缩成小专家。
Training a Student Expert via Semi-Supervised Foundation Model Distillation
- 通过自训练和对比校准实现领域适配，利用无标注数据提升性能。
- 学生模型比零样本教师模型高11.9和8.6个点的实例分割准确率。
- 适合资源受限场景下高效部署视觉基础模型的开发者。
视觉基础模型（VFMs）具有强大感知能力，但计算开销大且适配需昂贵标注。本文提出半监督知识蒸馏（SSKD）框架，仅用少量标注数据和大量无标注数据，将预训练视觉基础模型压缩为紧凑专家，应用于像素级标注成本高的实例分割任务。框架分三阶段：(1) 通过带对比校准的自训练进行领域适应；(2) 采用统一多目标损失实现知识迁移；(3) 学生模型精炼以缓解伪标签偏差。核心是实例感知的像素级对比损失，融合掩码与类别分数提取有信息量的负样本，并强化实例间边界。通过在适应与蒸馏中保持对比信号，对齐师生嵌入，更有效利用无标注图像。在Cityscapes和ADE20K上，约11倍压缩的学生模型，相比零样本教师模型分别提升11.9和8.6个点的AP，超过适配后教师模型3.4和1.5个点的AP，优于当前最优的半监督蒸馏方法。
原文摘要 · Abstract (English)
Foundation models deliver strong perception but are often too computationally heavy to deploy, and adapting them typically requires costly annotations. We introduce a semi-supervised knowledge distillation (SSKD) framework that compresses pre-trained vision foundation models (VFMs) into compact experts using limited labeled and abundant unlabeled data, and instantiate it for instance segmentation where per-pixel labels are particularly expensive. The framework unfolds in three stages: (1) domain adaptation of the VFM(s) via self-training with contrastive calibration, (2) knowledge transfer through a unified multi-objective loss, and (3) student refinement to mitigate residual pseudo-label bias. Central to our approach is an instance-aware pixel-wise contrastive loss that fuses mask and class scores to extract informative negatives and enforce clear inter-instance margins. By maintaining this contrastive signal across both adaptation and distillation, we align teacher and student embeddings and more effectively leverage unlabeled images. On Cityscapes and ADE20K, our $\approx 11\times$ smaller student improves over its zero-shot VFM teacher(s) by +11.9 and +8.6 AP, surpasses adapted teacher(s) by +3.4 and +1.5 AP, and outperforms state-of-the-art SSKD methods on benchmarks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。