用少量数据将单模态特征映射到多模态空间，性能超越CLIP。
CSA: Data-efficient Mapping of Unimodal Features to Multimodal Features
- 基于双单模态编码器和新相似度分数，构建多模态空间映射。
- 仅需CLIP所需数据的1/50,000即可达到更好性能。
- 适用于图像-文本外的模态对，如激光雷达与文本。
多模态编码器如CLIP在零样本图像分类和跨模态检索等任务中表现优异，但需大量训练数据。我们提出规范相似性分析（CSA），利用两个单模态编码器，在有限数据下复现多模态编码器。CSA通过新相似度分数将单模态特征映射至多模态空间，仅保留多模态信息。CSA仅涉及单模态编码器推理和立方复杂度矩阵分解，无需大规模GPU训练。实验表明，在预训练的ImageNet分类和误导性新闻标题检测任务中，给定预训练单模态编码器，CSA以50,000倍更少的多模态数据对实现性能超越CLIP。CSA优于现有最先进方法，可扩展至图像-文本之外的模态组合，如激光雷达与文本，为缺乏配对多模态数据但拥有丰富单模态数据的场景提供可行路径。
原文摘要 · Abstract (English)
Multimodal encoders like CLIP excel in tasks such as zero-shot image classification and cross-modal retrieval. However, they require excessive training data. We propose canonical similarity analysis (CSA), which uses two unimodal encoders to replicate multimodal encoders using limited data. CSA maps unimodal features into a multimodal space, using a new similarity score to retain only the multimodal information. CSA only involves the inference of unimodal encoders and a cubic-complexity matrix decomposition, eliminating the need for extensive GPU-based model training. Experiments show that CSA outperforms CLIP while requiring $50,000\times$ fewer multimodal data pairs to bridge the modalities given pre-trained unimodal encoders on ImageNet classification and misinformative news caption detection. CSA surpasses the state-of-the-art method to map unimodal features to multimodal features. We also demonstrate the ability of CSA with modalities beyond image and text, paving the way for future modality pairs with limited paired multimodal data but abundant unpaired unimodal data, such as lidar and text.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。