用生成式大模型直接教判别式模型，提升视觉语言理解能力。
MLLMCLIP: Feature-Level Distillation of MLLM for Robust Vision-Language Representations
- 从生成式多模态大模型蒸馏特征，跳过复杂数据生成流程。
- 在组合性任务上达到最新最好效果，零样本分类与图文检索均提升。
- 适合想提升模型推理能力但不想搞数据流水线的研究者。
预训练视觉语言模型如CLIP在零样本识别上表现优异，但在组合性理解方面表现不佳，尤其在属性-物体和关系结构上。近期研究通过级联大语言模型和文生图模型生成合成难样本进行训练，但带来巨大流程开销。本文提出MLLMCLIP，一种异构蒸馏框架，将生成式多模态大模型（MLLM）教师的多模态知识直接蒸馏到判别式CLIP学生模型中，完全绕过合成数据生成。为解决两者架构差异，引入基于注意力的逐层标记选择和基于CKA的蒸馏损失。相比已有CLIP增强方法，MLLMCLIP在组合性准确率上达到当前最优，并在标准零样本分类和图像-文本检索任务上持续提升，表明特征级蒸馏能同时增强组合性与通用视觉语言表征能力。
原文摘要 · Abstract (English)
Pretrained vision-language models such as CLIP excel at zero-shot recognition but often fail at compositionality, particularly attribute-object and relational structures. Recent studies mitigate this issue by augmenting training with synthetic hard negatives generated by a cascade of large language models and text-to-image models, which incurs substantial pipeline overhead. We instead propose MLLMCLIP, a heterogeneous distillation framework that transfers multimodal knowledge directly from a generative Multimodal Large Language Model (MLLM) teacher into a discriminative CLIP student, bypassing synthetic data entirely. To bridge the architectural mismatch between the two paradigms, we introduce an attention-based per-layer token selection and a CKA-based distillation loss. Compared to prior CLIP-enhancement methods, MLLMCLIP achieves state-of-the-art compositional accuracy while delivering consistent gains on standard zero-shot classification and image-text retrieval, showing that feature-level distillation strengthens both compositional and general vision-language representation capability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。