提出新模型评估缺失模态对预测的影响,提升不完整多模态数据的利用效率。
Characterizing the Predictive Impact of Modalities with Supervised Latent-Variable Modeling
- 用潜在变量建模缺失模态,结合已知模态进行预测
- 在全部模态缺失时性能接近单模态基线,在全模态可用时媲美多模态基线
- 可逐样本量化模态缺失的影响,适合处理不完整多模态数据的研究者
尽管多模态大语言模型取得成功,现有方法通常假设训练和推理时所有模态都可用。实际中,多模态数据常因模态缺失、异步采集或部分样本缺模态而残缺。本文提出PRIMO,一种监督式潜变量填补模型,用于量化任意缺失模态在多模态学习中的预测影响。该模型利用所有可用训练样本(无论模态是否完整),通过潜变量捕捉缺失模态与已知模态在预测上下文中的关系。推理时,从学习到的缺失模态分布中采样多个样本,既获得边际预测分布,又分析每个实例中缺失模态对预测的影响。我们在合成XOR数据集、Audio-Vision MNIST及MIMIC-III(用于死亡率与ICD-9编码预测)上评估了PRIMO。结果表明:当某模态完全缺失时,其性能接近单模态基线;当所有模态可用时,性能媲美多模态基线。通过方差度量,PRIMO实现了实例级的模态预测影响量化,并可视化了不同补全方式下可能的标签分布。
原文摘要 · Abstract (English)
Despite the recent success of Multimodal Large Language Models (MLLMs), existing approaches predominantly assume the availability of multiple modalities during training and inference. In practice, multimodal data is often incomplete because modalities may be missing, collected asynchronously, or available only for a subset of examples. In this work, we propose PRIMO, a supervised latent-variable imputation model that quantifies the predictive impact of any missing modality within the multimodal learning setting. PRIMO enables the use of all available training examples, whether modalities are complete or partial. Specifically, it models the missing modality through a latent variable that captures its relationship with the observed modality in the context of prediction. During inference, we draw many samples from the learned distribution over the missing modality to both obtain the marginal predictive distribution (for the purpose of prediction) and analyze the impact of the missing modalities on the prediction for each instance. We evaluate PRIMO on a synthetic XOR dataset, Audio-Vision MNIST, and MIMIC-III for mortality and ICD-9 prediction. Across all datasets, PRIMO obtains performance comparable to unimodal baselines when a modality is fully missing and to multimodal baselines when all modalities are available. PRIMO quantifies the predictive impact of a modality at the instance level using a variance-based metric computed from predictions across latent completions. We visually demonstrate how varying completions of the missing modality result in a set of plausible labels.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。