arXiv:2505.16953cs.LGstat.ML2025-05ICLR被引 1

忽略模态缺失会高估多模态信息价值,该研究提出纠正方法。

ICYM2I: The illusion of multimodal informativeness under missingness

  • 通过逆概率加权修正缺失模态带来的偏差
  • 在真实与合成数据上验证了不修正会导致信息增益误判
  • 适合关注多模态模型泛化性能的研究者

多模态学习在人工智能应用中持续受关注,因其有望通过融合不同数据模态实现信息增益。然而,源环境与目标环境中可观测的模态可能因成本、硬件故障或对某模态“信息量”的主观判断而不同,这种缺失模式的变化尚未被充分研究。若在未考虑缺失性的情况下直接估算新增模态的信息增益,可能导致目标环境中模态价值的错误评估。本文形式化了缺失性问题,揭示其普遍性,并证明当缺失过程未被显式建模时,会引发分布偏移与偏差。为此,提出ICYM2I(In Case You Multimodal Missed It)框架,基于逆概率加权对缺失情形下的预测性能与信息增益进行校正。在合成、半合成及真实数据集上均验证了该调整的重要性。

原文摘要 · Abstract (English)

Multimodal learning is of continued interest in artificial intelligence-based applications, motivated by the potential information gain from combining different data modalities. However, modalities observed in the source environment may differ from the modalities observed in the target environment due to multiple factors, including cost, hardware failure, or the perceived \textit{informativeness} of a given modality. This change in missingness patterns between the source and target environment has not been carefully studied. Na{ï}ve estimation of the information gain associated with including an additional modality without accounting for missingness may result in improper estimates of that modality's value in the target environment. We formalize the problem of missingness, demonstrate its ubiquity, and show that the subsequent distribution shift induces bias when the missingness process is not explicitly accounted for. To address this issue, we introduce ICYM2I (In Case You Multimodal Missed It), a framework for the evaluation of predictive performance and information gain under missingness through inverse probability weighting-based correction. We demonstrate the importance of the proposed adjustment to estimate information gain under missingness on synthetic, semi-synthetic, and real-world datasets.

多模态学习缺失数据信息增益逆概率加权

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。