用少量标注数据实现跨模态模型对齐，提升多模态任务性能。
Model alignment using inter-modal bridges
- 基于条件流匹配构建跨模态潜在空间桥梁，实现文本到图像等对齐。
- 在少于20%标注数据下，性能媲美端到端训练模型。
- 适用于标注稀缺场景，适合需要跨模态复用的开发者。
基础模型在语言和视觉等模态上表现卓越，但跨模态（如文本与视觉）的模型复用受限于内部表示对齐难题。现有方法依赖大量配对数据或局限于特定领域。本文提出一种半监督的跨模态对齐方法，基于条件流匹配学习不同模态间（如文本到图像、生物到人工神经活动）的潜在空间映射。该方法在两种设置下有效：(1) 通过带跨空间桥接代价的最优传输问题求解；(2) 利用标记样本进行内存高效对齐。尽管受原始模型容量限制，该方法在MNIST、ImageNet及 extcite{majaj2015simple}数据集上的物体识别与图像生成任务中，下游性能接近端到端训练模型，尤其在标注数据少于20%时表现突出。本方法为跨模态对齐提供了低监督、高效率的解决方案。
原文摘要 · Abstract (English)
Foundation models have demonstrated remarkable performance across modalities such as language and vision. However, model reuse across distinct modalities (e.g., text and vision) remains limited due to the difficulty of aligning internal representations. Existing methods require extensive paired training data or are constrained to specific domains. We introduce a semi-supervised approach for model alignment via conditional flow matching. The conditional flow between latent spaces of different modalities (e.g., text-to-image or biological-to-artificial neuronal activity) can be learned in two settings: ($1$) solving a (balanced or unbalanced) optimal transport problem with an inter-space bridge cost, and ($2$) performing memory-efficient alignment using labelled exemplars. Despite being constrained by the original models' capacity, our method--under both settings--matches downstream task performance of end-to-end trained models on object recognition and image generation tasks across MNIST, ImageNet, and \cite{majaj2015simple} datasets, particularly when labelled training data is scarce ($<20\%$). Our method provides a data-efficient solution for inter-modal model alignment with minimal supervision.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。