用词典扩展文本嵌入,让图像分类模型学得更准更鲁棒。
Crossmodal Knowledge Distillation with WordNet-Relaxed Text Embeddings for Robust Image Classification
- 用WordNet扩展类别名生成更丰富的文本提示,避免直接使用标签
- 在6个数据集上达到顶尖或第二好效果,提升学生模型性能
- 减少文本捷径依赖,增强视觉特征利用,适合图像分类任务
跨模态知识蒸馏旨在通过多模态教师模型提升单模态学生模型。当教师包含学生所用模态时,可引入额外互补信息以增强知识迁移。在监督图像分类中,图像数据集通常包含代表高层概念的类别标签,为引入文本线索提供了自然途径。然而,这些标签往往无法捕捉真实视觉中的深层语义结构,若直接作为输入则可能导致标签泄露,最终限制蒸馏性能。为此,我们提出一种多教师跨模态知识蒸馏框架,将CLIP图像嵌入与可学习的WordNet松弛文本嵌入结合,采用分层损失函数。通过避免直接使用精确类别名称,转而使用语义更丰富的WordNet扩展,缓解标签泄露问题,并引入更多样化的文本提示。实验表明,该策略显著提升学生模型性能,而噪声过大或过于精确的文本嵌入会降低蒸馏效率。可解释性分析显示,WordNet松弛提示促使模型更依赖视觉特征而非文本捷径,同时有效融合新引入的文本信息。本方法在六个公开数据集上实现当前最优或次优表现,证明其在推进跨模态知识蒸馏方面的有效性。
原文摘要 · Abstract (English)
Crossmodal knowledge distillation (KD) aims to enhance a unimodal student using a multimodal teacher model. In particular, when the teacher's modalities include the student's, additional complementary information can be exploited to improve knowledge transfer. In supervised image classification, image datasets typically include class labels that represent high-level concepts, suggesting a natural avenue to incorporate textual cues for crossmodal KD. However, these labels rarely capture the deeper semantic structures in real-world visuals and can lead to label leakage if used directly as inputs, ultimately limiting KD performance. To address these issues, we propose a multi-teacher crossmodal KD framework that integrates CLIP image embeddings with learnable WordNet-relaxed text embeddings under a hierarchical loss. By avoiding direct use of exact class names and instead using semantically richer WordNet expansions, we mitigate label leakage and introduce more diverse textual cues. Experiments show that this strategy significantly boosts student performance, whereas noisy or overly precise text embeddings hinder distillation efficiency. Interpretability analyses confirm that WordNet-relaxed prompts encourage heavier reliance on visual features over textual shortcuts, while still effectively incorporating the newly introduced textual cues. Our method achieves state-of-the-art or second-best results on six public datasets, demonstrating its effectiveness in advancing crossmodal KD.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。