任意模态组合的多模态表示学习框架,自适应融合且高效。
CLARGA: Multimodal Graph Representation Learning over Arbitrary Sets of Modalities
- 基于注意力图构建动态融合网络,按样本生成独特连接结构。
- 在7个数据集上超越主流模型,支持缺失模态输入仍保持性能。
- 适合多模态任务、模态不全或需灵活扩展的场景使用。
我们提出CLARGA,一种通用的多模态融合架构,可处理任意数量和类型的模态而无需修改基础框架。给定监督数据集,该方法能应用于几乎任何机器学习任务,通过下游层融合不同模态表示。每个样本独立构建注意力加权图,并利用多头图注意力网络在图上传递信息,实现自适应融合。该机制具有次二次复杂度,随模态数增长仍保持高效。通过可学习掩码,模型可适应缺失模态输入。训练采用混合目标:监督任务损失与对比式InfoNCE损失相结合,提升跨模态一致性与对噪声输入的鲁棒性。我们在涵盖金融、人机交互、通用多媒体分类及情感计算的7个数据集上验证其有效性,持续优于基线模型、最先进方法及消融实验。额外实验显示其对缺失输入的鲁棒性以及在小众任务中的表现优势。总体而言,CLARGA可无缝集成至各类模型中,实现多样任务下的高效表示学习。
原文摘要 · Abstract (English)
We introduce CLARGA, a general-purpose multimodal fusion architecture for multimodal representation learning that works with any number and type of modalities without changing the underlying framework. Given a supervised dataset, CLARGA can be applied to virtually any machine learning task to fuse different multimodal representations for processing by downstream layers. On a sample-by-sample basis, CLARGA learns how modalities should inform one another by building an attention weighted graph over their features and passing messages along this graph with a multi-head Graph Attention Network. Not only does this make CLARGA highly adaptive, as it constructs unique graphs for different samples, it makes for efficient fusion with sub-quadratic complexity as the number of modalities grows. Through a learnable mask, it can also adapt to missing modality inputs. The model is trained with a hybrid objective that combines a supervised task loss with contrastive InfoNCE loss, improving cross-modal consistency and robustness to noisy inputs. We demonstrate CLARGA's effectiveness in diverse multimodal representation learning tasks across 7 datasets spanning finance, human-computer interaction, general multimedia classification, and affective computing. It consistently outperforms baselines, state-of-the-art models, and ablations. Additional experiments also demonstrate its robustness to missing inputs and ability to excel on niche tasks. Overall, CLARGA can be easily plugged into machine learning models for effective and efficient learning of representations across a wide variety of tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。