arXiv:2511.21331cs.CVcs.AI2025-11中稿 · CVPR被引 3

提出对比融合框架,同时对齐多模态组合与成对关系。

The More, the Merrier: Contrastive Fusion for Higher-Order Multimodal Alignment

  • 引入融合对比损失,联合优化单模态与组合模态表示。
  • 在合成与真实数据集上实现检索与分类竞争力,支持多种检索模式。
  • 适合需捕捉复杂跨模态依赖的多模态任务,如视频理解、图文生成。

多模态表示学习仍是核心挑战。现有方法多为成对对齐,虽有研究尝试建模多模态高阶交互,但常忽略或弱化成对关系,影响单模态任务表现。本文提出对比融合(ConFu)框架,将单模态及其组合统一嵌入同一表示空间,使模态与其融合版本相互对齐。通过扩展传统成对对比目标,加入融合模态对比项,促使模态对与第三模态联合嵌入,从而捕捉无法仅通过成对对齐恢复的高阶依赖(如异或类关系),同时保持强成对对应性。我们在合成与真实世界多模态基准上评估了ConFu,测试其利用跨模态互补性、建模高阶依赖及随模态复杂度增长的可扩展性。结果表明,ConFu在检索与分类任务中表现优异,且在单一对比框架内支持一对一与两对一检索。代码与数据集已开源。

原文摘要 · Abstract (English)

Learning joint representations across multiple modalities remains a central challenge in multimodal machine learning. Prevailing approaches predominantly operate in pairwise settings, aligning two modalities at a time. While some recent methods aim to capture higher-order interactions among multiple modalities, they often overlook or insufficiently preserve pairwise relationships, limiting their effectiveness on single-modality tasks. In this work, we introduce Contrastive Fusion (ConFu), a framework that jointly embeds both individual modalities and their fused combinations into a unified representation space, where modalities and their fused counterparts are aligned. ConFu extends traditional pairwise contrastive objectives with an additional fused-modality contrastive term, encouraging the joint embedding of modality pairs with a third modality. This formulation enables ConFu to capture higher-order dependencies, such as XOR-like relationships, that cannot be recovered through pairwise alignment alone, while still maintaining strong pairwise correspondence. We evaluate ConFu on synthetic and real-world multimodal benchmarks, assessing its ability to exploit cross-modal complementarity, capture higher-order dependencies, and scale with increasing multimodal complexity. Across these settings, ConFu demonstrates competitive performance on retrieval and classification tasks, while supporting unified one-to-one and two-to-one retrieval within a single contrastive framework. We release our code and dataset at https://github.com/estafons/confu.

多模态对比学习融合模型

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。