通过混合样本对比学习,捕捉多模态数据中的共享关系,提升分类性能。
Harnessing Shared Relations via Multimodal Mixup Contrastive Learning for Multimodal Classification
- 用模态间混合样本构造对比损失,学习跨模态共享关系。
- 在N24News、ROSMAP、BRCA上优于现有方法,Food-101表现相当。
- 适合需要跨模态协同建模的多模态分类任务。
深度多模态学习通过对比学习捕捉模态间的显式一一对应关系取得了显著成功。然而,真实数据中常存在超越简单成对关联的共享关系。本文提出M3CoL(Multimodal Mixup Contrastive Learning),通过基于混合样本的对比损失,将一模态的混合样本与另一模态的对应样本对齐,从而捕捉其间的共享关系。针对多模态分类任务,设计融合模块与单模态预测模块并行的框架,在训练中引入辅助监督,并结合提出的混合对比损失。在多个数据集(N24News、ROSMAP、BRCA、Food-101)上的大量实验表明,M3CoL能有效捕捉共享多模态关系并在不同领域间良好泛化。其在N24News、ROSMAP、BRCA上优于当前最优方法,且在Food-101上表现相当。该工作凸显了学习共享关系对鲁棒多模态学习的重要性,为未来研究开辟新方向。代码已公开于https://github.com/RaghavSinghal10/M3CoL。
原文摘要 · Abstract (English)
Deep multimodal learning has shown remarkable success by leveraging contrastive learning to capture explicit one-to-one relations across modalities. However, real-world data often exhibits shared relations beyond simple pairwise associations. We propose M3CoL, a Multimodal Mixup Contrastive Learning approach to capture nuanced shared relations inherent in multimodal data. Our key contribution is a Mixup-based contrastive loss that learns robust representations by aligning mixed samples from one modality with their corresponding samples from other modalities thereby capturing shared relations between them. For multimodal classification tasks, we introduce a framework that integrates a fusion module with unimodal prediction modules for auxiliary supervision during training, complemented by our proposed Mixup-based contrastive loss. Through extensive experiments on diverse datasets (N24News, ROSMAP, BRCA, and Food-101), we demonstrate that M3CoL effectively captures shared multimodal relations and generalizes across domains. It outperforms state-of-the-art methods on N24News, ROSMAP, and BRCA, while achieving comparable performance on Food-101. Our work highlights the significance of learning shared relations for robust multimodal learning, opening up promising avenues for future research. Our code is publicly available at https://github.com/RaghavSinghal10/M3CoL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。