用大模型生成语义描述，增强CLIP图像编码器的分类能力。
LMM-Regularized CLIP Embeddings for Image Classification
- 用大模型提取图像语义，生成类别的平均描述
- 通过对比学习让图像嵌入靠近语义描述，提升判别力
- 适用于希望改进CLIP分类性能的研究者
本文针对图像分类任务，利用强大的CLIP多模态模型。目标是通过一种基于大型多模态模型（LMM）的正则化方法，提升基于CLIP图像编码器的分类性能。该方法首先使用LMM为数据集中的图像提取语义描述，再通过冻结的CLIP文本编码器获取对应文本嵌入，并计算各类别的均值语义描述。随后，对CLIP图像编码器添加分类头，联合训练图像编码器输出与分类头，在主分类目标外增加一个辅助目标：强制图像编码器输出的嵌入与对应的LMM生成的类别均值语义描述相似。此过程使嵌入具备更强的判别能力，从而提升分类性能。在三个图像分类数据集上的大量实验验证了该正则化方法的有效性。
原文摘要 · Abstract (English)
In this paper we deal with image classification tasks using the powerful CLIP vision-language model. Our goal is to advance the classification performance using the CLIP's image encoder, by proposing a novel Large Multimodal Model (LMM) based regularization method. The proposed method uses an LMM to extract semantic descriptions for the images of the dataset. Then, it uses the CLIP's text encoder, frozen, in order to obtain the corresponding text embeddings and compute the mean semantic class descriptions. Subsequently, we adapt the CLIP's image encoder by adding a classification head, and we train it along with the image encoder output, apart from the main classification objective, with an additional auxiliary objective. The additional objective forces the embeddings at the image encoder's output to become similar to their corresponding LMM-generated mean semantic class descriptions. In this way, it produces embeddings with enhanced discrimination ability, leading to improved classification performance. The effectiveness of the proposed regularization method is validated through extensive experiments on three image classification datasets.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。