arXiv:2604.05834cs.LG2026-04被引 1

多模态对比学习中,一个模态失效会破坏整体检索效果,新方法通过动态屏蔽不靠谱输入提升精度。

Hidden in the Multiplicative Interaction: Uncovering Fragility in Multimodal Contrastive Learning

论文配图:Hidden in the Multiplicative Interaction: Uncovering Fragility in Multimodal Contrastive Learning
图 1 · 摘自论文原文
  • 用注意力门控机制按样本动态调整各模态贡献度
  • 在合成数据和三个真实三模态数据集上,顶1准确率超越现有最佳模型
  • 适合处理噪声、错位或缺失模态的高阶多模态任务

对比学习已成为配对数据无监督学习的标准方法,如图像-文本匹配中的CLIP。然而,许多领域涉及超过两种模态,需捕捉超越成对对齐的高阶依赖关系。Symile通过将点积替换为模态嵌入间的多重线性内积(MIP)拓展了CLIP至该场景。本文揭示了一种隐藏于乘法交互中的脆弱性:单一弱信息、错位或缺失的模态可通过目标函数传播并扭曲跨模态检索分数。为此,我们提出Gated Symile,一种基于注意力的对比门控机制,可对每个候选样本自适应调整模态贡献。该门控通过插值嵌入至可学习的中性方向,在可靠跨模态对齐可能性低时引入显式空值(NULL)选项以抑制不可靠输入。在可控的合成基准及三个真实世界三模态数据集上,Gated Symile在调优后均优于当前最优基线,取得更高顶1检索准确率。更广泛地,结果强调了门控机制对于应对噪声、错位或缺失输入的鲁棒性多模态对比学习的重要性。

原文摘要 · Abstract (English)

Contrastive learning has become a standard approach for unsupervised learning from paired data, as demonstrated by CLIP for image-text matching. However, many domains involve more than two modalities and require objectives that capture higher-order dependencies beyond pairwise alignment. Symile extends CLIP to this setting by replacing the dot product with the multilinear inner product (MIP) over modality embeddings. In this work, we show that there is a fragility which ishidden in the multiplicative interaction: a single weakly informative, misaligned, or missing modality can propagate through the objective and distort cross-modal retrieval scores. We propose Gated Symile, a contrastive gating mechanism that adapts modality contributions on an attention-based, per-candidate basis. The gate suppresses unreliable inputs by interpolating embeddings toward learnable neutral directions with an explicit NULL option when reliable cross-modal alignment is unlikely. Across a controlled synthetic benchmark that uncovers this fragility and three real-world trimodal datasets, Gated Symile achieves higher top-1 retrieval accuracy than well-tuned state-of-the-art (sota) baselines. More broadly, our results highlight gating as a step toward robust multimodal contrastive learning beyond two modalities in the presence of noise, misalignment, or missing inputs.

多模态学习对比学习门控机制鲁棒性

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。