融合像素与特征空间的多目标知识蒸馏框架，提升图像模型性能。
MEDiC: Multi-objective Exploration of Distillation from CLIP
- 三重目标协同：像素重建、特征对齐与令牌蒸馏。
- 在ImageNet-1K上达到73.9% kNN准确率，微调精度达85.1%。
- 揭示损失权重敏感性，适合研究蒸馏机制与视觉模型优化者。
掩码图像建模（MIM）方法通常在原始像素空间（重建掩码块）或潜在特征空间（与预训练教师对齐）中运行。我们提出MEDiC（基于CLIP的多目标蒸馏探索），通过三个互补目标将两者结合：来自冻结CLIP编码器的块级令牌蒸馏、全局CLS对齐，以及轻量解码器实现的像素重建。我们系统研究了该多目标框架的设计空间。首先，三个目标提供互补信息，完整组合在ImageNet-1K上达到73.9% kNN准确率。其次，引入带相对位置偏置的层次聚类生成掩码，虽更语义连贯，但在教师引导蒸馏中未优于简单块掩码，归因于教师固有的语义感知能力。第三，发现最优标量损失权重极脆弱，微小扰动可导致kNN准确率下降高达17个百分点。该框架在ViT-Base上使用300个周期训练，实现73.9% kNN和85.1%微调准确率。
原文摘要 · Abstract (English)
Masked image modeling (MIM) methods typically operate in either raw pixel space (reconstructing masked patches) or latent feature space (aligning with a pre-trained teacher). We present MEDiC (Multi-objective Exploration of Distillation from CLIP), a framework that combines both spaces in a single pipeline through three complementary objectives: patch-level token distillation from a frozen CLIP encoder, global CLS alignment, and pixel reconstruction via a lightweight decoder. We conduct a systematic investigation of the design space surrounding this multi-objective framework. First, we show that all three objectives provide complementary information, with the full combination reaching 73.9% kNN accuracy on ImageNet-1K. Second, we introduce hierarchical clustering with relative position bias for evolved masking and find that, despite producing more semantically coherent masks than prior methods, evolved masking does not outperform simple block masking in the teacher-guided distillation setting, a finding we attribute to the teacher's inherent semantic awareness. Third, we reveal that optimal scalar loss weights are extremely fragile, with small perturbations causing drops of up to 17 percentage points in kNN accuracy. Our framework achieves 73.9% kNN and 85.1% fine-tuning accuracy with ViT-Base at 300 epochs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。