arXiv:2511.12034cs.CVcs.LG2025-11中稿 · ICML被引 6

解决多模态数据缺失时的对齐偏差问题,提升模型鲁棒性。

Calibrated Multimodal Representation Learning with Missing Modalities

  • 基于锚点偏移理论,设计可校准缺失模态的对齐方法
  • 在多个数据集上实现优于基线的对齐精度,最高提升4.2%
  • 适用于真实场景中常出现模态缺失的数据集,如医学影像与文本

多模态表示学习通过将不同模态对齐到统一潜在空间来实现融合。现有方法通常要求所有模态同时存在,难以处理普遍存在模态缺失的数据集。本文从锚点偏移视角揭示该问题:观测模态与理想全模态锚点存在偏差,导致不可避免的对齐偏移。为此,提出CalMRL方法,利用模态先验与内在关联,在表示层面建模缺失模态的补全。采用两步学习策略,获得共享潜在变量后验分布的闭式解,有效缓解锚点偏移并保障收敛性。通过将校准对齐集成至先进方法,显著扩展了对缺失数据的适用范围。大量实验验证其有效性,且代码已开源。

原文摘要 · Abstract (English)

Multimodal representation learning harmonizes distinct modalities by aligning them into a unified latent space. Recent research generalizes traditional cross-modal alignment to produce enhanced multimodal synergy but requires all modalities to be present for a common instance, making it challenging to utilize prevalent datasets with missing modalities. We provide theoretical insights into this issue from an anchor shift perspective. Observed modalities are aligned with a local anchor that deviates from the optimal one when all modalities are present, resulting in an inevitable shift. To address this, we propose CalMRL to calibrate incomplete alignments caused by missing modalities. CalMRL leverages the priors and the inherent connections among modalities to model the imputation for the missing ones at the representation level. To resolve the optimization dilemma, we employ a bi-step learning method with the closed-form solution of the posterior distribution of shared latents. We validate its mitigation of anchor shift and convergence with theoretical guidance. By equipping the calibrated alignment with the existing advanced method, we offer new flexibility to absorb data with missing modalities, which is originally unattainable. Extensive experiments demonstrate the superiority of CalMRL. The code is released at https://github.com/Xiaohao-Liu/CalMRL.

多模态学习表示对齐缺失数据校准

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。