提出完美多模态对齐方法，提升跨模态表示一致性
Towards Achieving Perfect Multimodal Alignment
- 将多模态对齐建模为逆问题，通过SVD实现近似完美对齐
- 在合成高斯数据上优于对比学习对齐方法，准确率显著提升
- 适用于动作识别等跨模态任务，可推广至多种模态与场景
多模态对齐旨在构建一个联合隐空间，使代表同一概念的不同模态映射到邻近的隐向量。本文将其形式化为一个逆问题，并证明在特定条件下，来自各模态的配对数据可映射至相同的隐向量，称为完美对齐。当无法实现完美对齐时，可通过多模态数据矩阵的奇异值分解（SVD）进行逼近。在合成多模态高斯数据上的实验验证了该方法的有效性，相较于学习型对比对齐方法表现更优。进一步在人体动作识别任务中展示了跨模态迁移的应用效果，完美对齐显著提升了模型精度。文章最后讨论了该方法在多种模态与任务中的应用潜力及其局限性，期望激发更多关于完美对齐及其表示学习应用的研究。
原文摘要 · Abstract (English)
Multimodal alignment constructs a joint latent vector space where modalities representing the same concept map to neighboring latent vectors. We formulate this as an inverse problem and show that, under certain conditions, paired data from each modality can map to equivalent latent vectors, which we refer to as perfect alignment. When perfect alignment cannot be achieved, it can be approximated using the Singular Value Decomposition (SVD) of a multimodal data matrix. Experiments on synthetic multimodal Gaussian data verify the effectiveness of our perfect alignment method compared to a learned contrastive alignment method. We further demonstrate the practical application of cross-modal transfer for human action recognition, showing that perfect alignment significantly enhances the model's accuracy. We conclude by discussing how these findings can be applied to various modalities and tasks and the limitations of our method. We hope these findings inspire further exploration of perfect alignment and its applications in representation learning.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。