用大模型知识蒸馏提升CLIP图文对齐能力
Enhancing CLIP Conceptual Embedding through Knowledge Distillation
- 用Llama 2作为教师模型，蒸馏文本嵌入以增强编码器
- 通过聚类生成软概念标签，让模型学习更细粒度语义
- 三阶段联合训练，显著提升图文匹配准确率
近期，CLIP已成为多模态场景下图像与文本对齐的重要模型。然而研究发现，其图文编码器在捕捉图文对的细节知识方面存在局限。为此，本文提出Knowledge-CLIP，一种基于Llama 2的新型知识蒸馏方法，旨在提升CLIP性能。该方法包含三个目标：文本嵌入蒸馏、概念学习和对比学习。首先，通过训练知识蒸馏的文本编码器，使其模仿教师模型Llama 2；其次，利用离线K-means聚类对Llama 2的文本数据进行分析，为每个图文对分配软概念标签，使Knowledge-CLIP能从这些标签中学习；最后，通过对比学习对齐图文嵌入。实验表明，所提模型显著提升了文本与图像编码器的性能。
原文摘要 · Abstract (English)
Recently, CLIP has become an important model for aligning images and text in multi-modal contexts. However, researchers have identified limitations in the ability of CLIP's text and image encoders to extract detailed knowledge from pairs of captions and images. In response, this paper presents Knowledge-CLIP, an innovative approach designed to improve CLIP's performance by integrating a new knowledge distillation (KD) method based on Llama 2. Our approach focuses on three key objectives: Text Embedding Distillation, Concept Learning, and Contrastive Learning. First, Text Embedding Distillation involves training the Knowledge-CLIP text encoder to mirror the teacher model, Llama 2. Next, Concept Learning assigns a soft concept label to each caption-image pair by employing offline K-means clustering on text data from Llama 2, enabling Knowledge-CLIP to learn from these soft concept labels. Lastly, Contrastive Learning aligns the text and image embeddings. Our experimental findings show that the proposed model improves the performance of both text and image encoders.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。