提出无需锚点的多模态对齐框架,提升表示学习稳定性与统一性。
Principled Multimodal Representation Learning

- 基于主导奇异值优化实现多模态无锚对齐
- 在多个任务上超越基线方法,有效防止表示坍缩
- 适合需要统一多模态表征的研究者参考
多模态表示学习旨在通过融合多种数据模态构建统一的表示空间,以增强多模态理解能力。传统方法依赖成对对比学习,需预设锚点模态,限制了跨模态对齐。近期工作尝试同时对齐多个模态,但仍面临固定锚点限制及奇异值乘积优化带来的不稳定性。为此,本文提出原理性多模态表示学习(PMRL),一种无需锚点、更稳定的多模态同步对齐框架。其核心思想是:完全对齐对应于秩-1的格拉姆矩阵,因此通过优化表示矩阵的最大奇异值,使各模态沿共享主方向对齐。我们设计了一种基于softmax的损失函数,将奇异值视为logits以突出最大奇异值;同时,在主特征向量上引入实例级对比正则化,保持实例间可分性并防止表示坍缩。在多种任务上的大量实验表明,PMRL显著优于基线方法。源代码见https://github.com/Xiaohao-Liu/PMRL。
原文摘要 · Abstract (English)
Multimodal representation learning seeks to create a unified representation space by integrating diverse data modalities to improve multimodal understanding. Traditional methods often depend on pairwise contrastive learning, which relies on a predefined anchor modality, restricting alignment across all modalities. Recent advances have investigated the simultaneous alignment of multiple modalities, yet several challenges remain, such as limitations imposed by fixed anchor points and instability arising from optimizing the product of singular values. To address the challenges, in this paper, we propose Principled Multimodal Representation Learning (PMRL), a novel framework that achieves simultaneous alignment of multiple modalities without anchor dependency in a more stable manner. Specifically, grounded in the theoretical insight that full alignment corresponds to a rank-1 Gram matrix, PMRL optimizes the dominant singular value of the representation matrix to align modalities along a shared leading direction. We propose a softmax-based loss function that treats singular values as logits to prioritize the largest singular value. Besides, instance-wise contrastive regularization on the leading eigenvectors maintains inter-instance separability and prevents representation collapse. Extensive experiments across diverse tasks demonstrate PMRL's superiority compared to baseline methods. Source code can be found in https://github.com/Xiaohao-Liu/PMRL.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。