提升轻量模型对齐能力，让小模型更像大模型的视觉语言结构。
CLIP-RD: Relative Distillation for Efficient CLIP Knowledge Distillation
- 通过垂直与交叉关系蒸馏，建模师生特征间的多向依赖关系。
- 在多个数据集上超越现有方法0.8个百分点的准确率。
- 适合需要高效部署CLIP能力的工业场景或边缘设备应用。
CLIP通过对比学习对齐图像与文本嵌入，展现出强大的零样本泛化能力。其大规模架构带来高计算与内存开销，促使将知识蒸馏至轻量级学生模型。然而，现有蒸馏方法未显式建模教师与学生嵌入间的多方向关系依赖，限制了学生模型对教师编码结构关系的保留。为此，我们提出一种关系知识蒸馏框架，引入两种新方法：垂直关系蒸馏（VRD）和交叉关系蒸馏（XRD）。VRD 在分布层面强制跨模态教师-学生蒸馏强度一致，而XRD 强制跨模态师生相似度分布具有双向对称性。联合建模多方向关系结构，使学生嵌入几何结构更忠实于教师，性能优于现有方法0.8个百分点。
原文摘要 · Abstract (English)
CLIP aligns image and text embeddings via contrastive learning and demonstrates strong zero-shot generalization. Its large-scale architecture requires substantial computational and memory resources, motivating the distillation of its capabilities into lightweight student models. However, existing CLIP distillation methods do not explicitly model multi-directional relational dependencies between teacher and student embeddings, limiting the student's ability to preserve the structural relationships encoded by the teacher. To address this, we propose a relational knowledge distillation framework that introduces two novel methods, Vertical Relational Distillation (VRD) and Cross Relational Distillation (XRD). VRD enforces consistency of teacher-student distillation strength across modalities at the distribution level, while XRD imposes bidirectional symmetry on cross-modal teacher-student similarity distributions. By jointly modeling multi-directional relational structures, CLIP-RD promotes faithful alignment of the student embedding geometry with that of the teacher, outperforming existing methods by 0.8%p.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。