用高斯混合模型替代Softmax，提升CLIP/ImageBind嵌入空间的分类性能
Performance of Gaussian Mixture Model Classifiers on Embedded Feature Spaces
- 在CLIP/ImageBind嵌入空间中使用低参数GMM分类器
- 多数情况下单个高斯分量即可有效表征类别，性能接近或优于Softmax
- ImageBind在图像分类任务中表现优于CLIP，尤其在降维后仍保持优势
CLIP和ImageBind提供的数据嵌入为多模态数据分析提供了强大特征。本文评估了基于高斯混合模型（GMM）的分类器在这些嵌入空间中的分类表现，作为标准Softmax层的替代方案。近年来，端到端训练的深度学习管道中已证明GMM分类器具有优异性能。我们的第一项贡献是研究了利用CLIP与ImageBind嵌入空间的GMM分类性能；第二项是提出一种参数更少的新型GMM分类器。结果表明，在大多数测试场景中，每个类别仅需一个高斯分量即可有效捕捉特征，我们推测这可能源于对比学习训练导致的同类特征自然聚集。此外，即使在通过PCA压缩嵌入空间后，ImageBind在图像数据集分类任务中的表现仍普遍优于CLIP。
原文摘要 · Abstract (English)
Data embeddings with CLIP and ImageBind provide powerful features for the analysis of multimedia and/or multimodal data. We assess their performance here for classification using a Gaussian Mixture models (GMMs) based layer as an alternative to the standard Softmax layer. GMMs based classifiers have recently been shown to have interesting performances as part of deep learning pipelines trained end-to-end. Our first contribution is to investigate GMM based classification performance taking advantage of the embedded spaces CLIP and ImageBind. Our second contribution is in proposing our own GMM based classifier with a lower parameters count than previously proposed. Our findings are, that in most cases, on these tested embedded spaces, one gaussian component in the GMMs is often enough for capturing each class, and we hypothesize that this may be due to the contrastive loss used for training these embedded spaces that naturally concentrates features together for each class. We also observed that ImageBind often provides better performance than CLIP for classification of image datasets even when these embedded spaces are compressed using PCA.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。