用Mamba替代CLIP的Transformer，提升跨模态检索性能并降低计算开销。
CLIMP: Contrastive Language-Image Mamba Pretraining
- 全Mamba架构取代双编码器，捕捉视觉与语言序列结构。
- 在ImageNet-O上比CLIP-ViT-B高7.5%准确率，16倍分辨率下检索精度提升6.6%。
- 支持可变输入分辨率，内存减少5倍，推理效率更高，适合实际部署。
对比语言-图像预训练（CLIP）依赖于对伪相关性敏感且随分辨率平方增长的视觉变换器。为此，我们提出首个完全基于Mamba的对比视觉语言模型CLIMP，将视觉和文本编码器均替换为Mamba。新架构在视觉与语言中编码序列结构，其中VMamba捕获视觉空间归纳偏置，减少对伪相关性的依赖，并生成更利于跨模态检索和分布外鲁棒性的嵌入空间，在ImageNet-O上超越OpenAI的CLIP-ViT-B达7.5%。CLIMP无需位置编码插值或特殊训练即可自然支持可变输入分辨率，在16倍训练分辨率下实现最高6.6%的检索准确率提升，同时内存使用减少5倍，浮点运算量减少1.8倍。自回归文本编码器还克服了CLIP固定上下文的局限，支持密集字幕检索。结果表明，Mamba在视觉语言学习中具有显著优势，是Transformer-based CLIP的有力替代方案。代码与模型已公开。
原文摘要 · Abstract (English)
Contrastive Language-Image Pre-training (CLIP) relies on Vision Transformers whose attention mechanism is susceptible to spurious correlations, and scales quadratically with resolution. To address these limitations, We present CLIMP, the first fully Mamba-based contrastive vision-language model that replaces both the vision and text encoders with Mamba. The new architecture encodes sequential structure in both vision and language, with VMamba capturing visual spatial inductive biases, reducing reliance on spurious correlations and producing an embedding space favorable for cross-modal retrieval and out-of-distribution robustness-surpassing OpenAI's CLIP-ViT-B by 7.5% on ImageNet-O. CLIMP naturally supports variable input resolutions without positional encoding interpolation or specialized training, achieving up to 6.6% higher retrieval accuracy at 16x training resolution while using 5x less memory and 1.8x fewer FLOPs. The autoregressive text encoder further overcomes CLIP's fixed context limitation, enabling dense captioning retrieval. Our findings suggest that Mamba exhibits advantageous properties for vision-language learning, making it a compelling alternative to Transformer-based CLIP.The code and models are publicly available at https://github.com/NimrodShabtay/CLIMP}
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。