用语义桥接解决CLIP少样本分类的模态错位问题
SeMoBridge: Semantic Modality Bridge for Efficient Few-Shot Adaptation of CLIP
- 将图像映射到文本空间，保持语义不变
- 1-4样本下性能超越现有方法，训练时间更短
- 轻量级闭式解法，适合低资源场景
尽管对比语言图像预训练（CLIP）在零样本任务中表现优异，但其在少样本分类中的性能受限于模态内错位问题。该问题源于持续存在的模态差距以及仅依赖跨模态训练目标，导致嵌入空间未校准，使图像间直接比较不可靠。现有方法通过优化相似度得分或高成本的逐样本优化来缓解，但效果有限。本文提出SeMoBridge，一种轻量级但高效的解决方案，直接建立图像到文本模态的语义桥接，保持语义完整。该方法为闭式解，可选多模态监督，结合图像与文本对齐损失进行优化。实验表明，训练版SeMoBridge-T仅需少量训练时间，在1、2、4样本场景下整体优于其他方法。
原文摘要 · Abstract (English)
While Contrastive Language-Image Pretraining (CLIP) excels at zero-shot tasks by aligning image and text embeddings, its performance in few-shot classification is hindered by a critical limitation: intra-modal misalignment. This issue, caused by a persistent modality gap and CLIP's exclusively inter-modal training objective, leaves the embedding spaces uncalibrated, making direct image-to-image comparisons unreliable. Existing methods attempt to address this by refining similarity logits or by computationally expensive per-sample optimization. To overcome these challenges, we introduce SeMoBridge, a lightweight yet powerful approach that directly addresses the misalignment. Our method maps images into the text modality, while keeping their semantic content intact through what we call a Semantic Modality Bridge. SeMoBridge is closed-form and can optionally be trained through multi-modal supervision, combining image and text-alignment losses to optimize the projection. Experiments show that the trained version, SeMoBridge-T, requires only a fraction of the training time while overall outperforming other methods, particularly in low-data scenarios (1, 2, and 4 shots). The code is available at https://github.com/christti98/semobridge.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。