用线性映射+三元组损失缓解视觉与文本特征差异，提升少样本图像分类效果。
Cross-Modal Mapping: Mitigating the Modality Gap for Few-Shot Image Classification
- 通过线性变换和三元组损失对齐图像与文本特征空间。
- 在11个基准数据集上平均准确率提升1.06%，4个分布外数据集表现优异。
- 无需微调主干网络，适合快速部署于少样本学习场景。
少样本图像分类在数据稀缺环境下仍是计算机视觉的关键挑战。现有方法多依赖预训练的视觉-语言模型（如CLIP），但因图像与文本特征在联合嵌入空间中分布不一致（即模态差距），直接使用文本特征作为类别原型会导致性能不佳。为此，本文提出一种新的跨模态映射（Cross-Modal Mapping, CMM）方法：通过线性变换全局对齐图像特征与文本特征空间，并利用三元组损失优化其局部空间关系，显著提升跨模态一致性。实验表明，相比部分微调主干网络的方法，CMM 在11个基准数据集上的平均Top-1准确率提升1.06%，且在4个分布外数据集上表现突出。CMM有效缓解了预训练模型中的模态差距，使文本特征可作为图像特征的有效类别原型，为少样本学习提供了一种高效且高度泛化的解决方案。
原文摘要 · Abstract (English)
Few-shot image classification remains a critical challenge in the field of computer vision, particularly in data-scarce environments. Existing methods typically rely on pre-trained visual-language models, such as CLIP. However, due to the modality gap, which is the inconsistent distribution of image and text features in the joint embedding space, directly using these features as class prototypes often leads to suboptimal performance. To address this issue, we propose a novel Cross-Modal Mapping (CMM) method. This method globally aligns image features with the text feature space through linear transformation and optimizes their local spatial relationships using triplet loss, thereby significantly enhancing cross-modal consistency. Experimental results show that compared to other methods, CMM simplifies the training process and demonstrates higher efficiency. Furthermore, CMM improves the average Top-1 accuracy by 1.06% on 11 benchmark datasets compared to methods that partially fine-tune the backbone, and it performs excellently on 4 distribution shift datasets. Notably, CMM effectively mitigates the modality gap in pre-trained models, enabling text features to serve as effective class prototypes for image features, thus providing an efficient and highly generalizable solution for few-shot learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。