用统一编码器早期融合图文标记，实现更强大的多模态表示。
FuseLIP: Multimodal Embeddings via Early Fusion of Discrete Tokens
- 将图文标记合并到统一词汇表，用单个Transformer进行早期融合
- 在多模态和单模态任务上均超越传统后期融合方法
- 专为多模态输入设计，适合需要联合理解图文的场景
对比语言-图像预训练通过各自模态的独立编码器将文本-图像对特征对齐至共享潜在空间。尽管该方法在多个零样本任务中表现优异，但无法原生处理多模态输入，即无法将图像和文本编码为单一特征向量。为此，通常需额外模块融合由单模态编码器提取的特征。本文提出FuseLIP，一种新型多模态嵌入架构。基于离散图像标记化最新进展，我们建议使用单一Transformer模型，作用于文本与图像标记的统一词汇表。这种早期融合方式使不同模态在编码每一层深度都能交互，获得比常见后期融合更丰富的表示。我们构建了新的多模态预训练与评估数据集，并设计具有挑战性的任务来检验多模态编码器性能。实验表明，FuseLIP在多项多模态及单模态嵌入任务中优于后期融合方法。
原文摘要 · Abstract (English)
Contrastive language-image pre-training aligns features of text-image pairs in a common latent space via distinct encoders for each modality. While this approach achieves impressive performance in several zero-shot tasks, it cannot natively handle multimodal inputs, i.e., encoding image and text into a single feature vector. As a remedy, it is common practice to use additional modules to merge the features extracted by unimodal encoders. In this work, we present FuseLIP, a new architecture for multimodal embedding. Leveraging recent progress in discrete image tokenizers, we propose to use a single transformer model operating on a unified vocabulary of text and image tokens. This early fusion approach allows the different modalities to interact at each depth of encoding and obtain richer representations compared to common late fusion. We collect new datasets for multimodal pre-training and evaluation, designing challenging tasks for multimodal encoders. We show that FuseLIP outperforms late fusion approaches in several multimodal and unimodal embedding tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。