用两个教师模型提升视觉语言模型压缩效果，尤其在领域迁移时表现更优。
The Professor: Multi-Teacher Unsupervised Prompt Distillation for Vision-Language Models
- 采用领域微调与零样本双教师模型联合蒸馏。
- 在欧标数据集上提升平均命中率5.78点，显著优于单教师。
- 适合需要跨域泛化的轻量化视觉语言模型部署场景。
提示蒸馏通过匹配教师模型在无标签数据上的预测结果，将大型视觉语言模型（如CLIP）压缩为轻量级学生模型。PromptKD（CVPR 2024）首次提出该范式，使用一个经过PromptSRC微调的ViT-L/14教师和一个ViT-B/16学生。本文提出TheProfessor，一种多教师扩展方法，从固定双教师集成中蒸馏：一个领域微调的PromptSRC ViT-L/14教师和一个零样本EVA-CLIP-L/14教师，其输出logits预先计算好。我们在四个基线到新类数据集（Caltech-101、DTD、UCF101、EuroSAT）上评估了单教师PromptKD、等概率集成和置信度加权集成。在12次单种子实验中，置信度加权集成使平均命中率（HM）从87.52提升至89.28（+1.77），等平均集成提升至88.88（+1.37）。增益具有数据集依赖性：在Caltech-101上几乎无提升（+0.16），在UCF101上适度提升（+0.62），在领域偏移的EuroSAT上最大（+5.78）。结果更新了此前仅基于Caltech的分析，表明多教师提示蒸馏在第二教师提供互补监督时最有效。
原文摘要 · Abstract (English)
Prompt distillation compresses large vision-language models (VLMs) such as CLIP into lightweight student models by matching teacher predictions on unlabeled domain images. PromptKD (CVPR 2024) established this paradigm with a single PromptSRC-finetuned ViT-L/14 teacher and a ViT-B/16 student. We propose TheProfessor, a multi-teacher extension that distills from a fixed two-teacher ensemble: a domain-finetuned PromptSRC ViT-L/14 teacher and a zero-shot EVA-CLIP-L/14 teacher whose logits are pre-computed per dataset. We evaluate single-teacher PromptKD, equal-probability ensembling, and confidence-weighted ensembling on four base-to-novel datasets: Caltech-101, DTD, UCF101, and EuroSAT. In a 12-run single-seed sweep, confidence-weighted ensembling improves average HM from 87.52 to 89.28 (+1.77 points), while equal averaging improves average HM to 88.88 (+1.37 points). Gains are dataset dependent: they are negligible on Caltech-101 (+0.16 HM for confidence weighting), modest on UCF101 (+0.62), and largest on domain-shifted EuroSAT (+5.78). These results update our earlier Caltech-only analysis and show that multi-teacher prompt distillation is most useful when the second teacher contributes complementary supervision under domain shift.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。