用柯西-施瓦茨散度实现视觉语言分布对齐，提升跨模态匹配精度
Distributional Vision-Language Alignment by Cauchy-Schwarz Divergence
- 引入柯西-施瓦茨散度，同时优化模态间对齐与分布均匀性
- 在图文生成和跨模态检索任务上显著优于基线方法
- 适合需要细粒度对齐的多模态应用开发者
视觉-语言对齐对跨模态生成与检索等下游任务至关重要。现有方法如CLIP使用InfoNCE最大化互信息，主要聚焦成对样本对齐，忽略模态间分布差异。此外，InfoNCE在对齐与均匀性之间存在固有矛盾，导致模态间隙下对齐效果不佳。为此，本文提出CS-Aligner框架，通过结合柯西-施瓦茨（CS）散度与互信息，实现分布级视觉-语言对齐。该方法同时捕捉各模态全局分布与成对语义关系。实验表明，CS散度可无缝解决InfoNCE的对齐-均匀性冲突，与之互补，实现更紧密精确的对齐。此外，引入分布对齐机制后，可利用未配对数据与词级别表示，增强实际应用中的灵活细粒度对齐能力。在文本到图像生成与跨模态检索任务上的实验验证了该方法的有效性。
原文摘要 · Abstract (English)
Vision-language alignment is crucial for various downstream tasks such as cross-modal generation and retrieval. Previous multimodal approaches like CLIP utilize InfoNCE to maximize mutual information, primarily aligning pairwise samples across modalities while overlooking distributional differences. In addition, InfoNCE has inherent conflict in terms of alignment and uniformity in multimodality, leading to suboptimal alignment with modality gaps. To overcome the limitations, we propose CS-Aligner, a novel framework that performs distributional vision-language alignment by integrating Cauchy-Schwarz (CS) divergence with mutual information. CS-Aligner captures both the global distribution information of each modality and the pairwise semantic relationships. We find that the CS divergence seamlessly addresses the InfoNCE's alignment-uniformity conflict and serves complementary roles with InfoNCE, yielding tighter and more precise alignment. Moreover, by introducing distributional alignment, CS-Aligner enables incorporating additional information from unpaired data and token-level representations, enhancing flexible and fine-grained alignment in practice. Experiments on text-to-image generation and cross-modality retrieval tasks demonstrate the effectiveness of our method on vision-language alignment.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。