提出一种统一方法,让模型在任意模态缺失时仍保持稳定性能。
MMP: Towards Robust Multi-Modal Learning with Masked Modality Projection
- 训练时随机屏蔽部分模态,学习用现有模态重建缺失模态的表示。
- 在多个数据集上验证,对不同缺失场景的鲁棒性显著优于现有方法。
- 适用于任意组合的多模态输入,无需为每种组合重新训练。
多模态学习旨在融合多种输入源数据以提升下游任务性能。但在真实场景中,若某些模态缺失,性能可能大幅下降。现有方法需针对每种模态组合进行定制训练或适配,既受限于特定模态,又随模态数量增加而计算开销剧增。本文提出掩码模态投影(MMP),通过在训练时随机屏蔽部分模态,学习将可用模态映射以估计被屏蔽模态的特征表示,从而实现单一模型对任意缺失模态场景的鲁棒性。实验在多个基线模型与数据集上验证该策略的有效性,结果表明,MMP在不同缺失场景下均优于现有针对缺失模态或特定模态组合设计的方法。
原文摘要 · Abstract (English)
Multimodal learning seeks to combine data from multiple input sources to enhance the performance of different downstream tasks. In real-world scenarios, performance can degrade substantially if some input modalities are missing. Existing methods that can handle missing modalities involve custom training or adaptation steps for each input modality combination. These approaches are either tied to specific modalities or become computationally expensive as the number of input modalities increases. In this paper, we propose Masked Modality Projection (MMP), a method designed to train a single model that is robust to any missing modality scenario. We achieve this by randomly masking a subset of modalities during training and learning to project available input modalities to estimate the tokens for the masked modalities. This approach enables the model to effectively learn to leverage the information from the available modalities to compensate for the missing ones, enhancing missing modality robustness. We conduct a series of experiments with various baseline models and datasets to assess the effectiveness of this strategy. Experiments demonstrate that our approach improves robustness to different missing modality scenarios, outperforming existing methods designed for missing modalities or specific modality combinations.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。