用无配对多模态数据提升单一模态模型性能
Better Together: Leveraging Unpaired Multimodal Data for Stronger Unimodal Models
- 共享参数的交替训练，不依赖数据配对
- 在图像、音频任务中均显著提升下游效果
- 适合缺乏配对数据的单模态学习场景
传统多模态学习依赖成对数据来寻找统一表征，但一个被忽视却潜力巨大的问题是：能否利用辅助的无配对多模态数据直接增强目标模态的表征学习？我们提出UML（Unpaired Multimodal Learner），一种与模态无关的训练范式，通过在不同模态间交替处理输入并共享参数，假设各模态是共享底层现实的投影，从而在无需显式配对的情况下利用跨模态结构。理论上，在线性数据生成假设下，无配对辅助数据可提供比单模态训练更丰富的数据生成过程表征。实证上，使用来自文本、音频或图像等辅助模态的无配对数据，持续提升了图像和音频等多样化单模态任务的下游性能。
原文摘要 · Abstract (English)
Traditional multimodal learners find unified representations for tasks like visual question answering, but rely heavily on paired datasets. However, an overlooked yet potentially powerful question is: can one leverage auxiliary unpaired multimodal data to directly enhance representation learning in a target modality? We introduce UML: Unpaired Multimodal Learner, a modality-agnostic training paradigm in which a single model alternately processes inputs from different modalities while sharing parameters across them. This design exploits the assumption that different modalities are projections of a shared underlying reality, allowing the model to benefit from cross-modal structure without requiring explicit pairs. Theoretically, under linear data-generating assumptions, we show that unpaired auxiliary data can yield representations strictly more informative about the data-generating process than unimodal training. Empirically, we show that using unpaired data from auxiliary modalities -- such as text, audio, or images -- consistently improves downstream performance across diverse unimodal targets such as image and audio. Our project page: https://unpaired-multimodal.github.io/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。