arXiv:2602.23353cs.LGcs.AI2026-02被引 1

用少量图文对+大量无配对数据,实现视觉语言模型高效对齐。

SOTAlign: Semi-Supervised Alignment of Unimodal Vision and Language Models via Optimal Transport

  • 先用少量配对数据学粗略共享结构,再用最优传输优化无配对数据
  • 在多个数据集上超越监督与半监督基线,性能显著提升
  • 适合资源有限但有海量单模态数据的场景

柏拉图表征假说认为不同模态的神经网络会收敛到共同的世界统计模型。近期工作通过轻量级对齐层将冻结的预训练视觉与语言模型对齐,但通常依赖对比损失和数百万对图文数据。本文提出一种半监督设置:仅用少量图像-文本配对样本,结合大量无配对数据,对预训练的单模态编码器进行对齐。我们提出SOTAlign框架,第一阶段使用线性教师从有限配对数据中恢复粗粒度共享几何结构;第二阶段基于最优传输的散度,在无配对样本上精炼对齐,传递关系结构而不过度约束目标空间。SOTAlign有效利用无配对图像与文本,学习跨数据集和编码器对的鲁棒联合嵌入,显著优于监督与半监督基线。代码已开源。

原文摘要 · Abstract (English)

The Platonic Representation Hypothesis posits that neural networks trained on different modalities converge toward a shared statistical model of the world. Recent work exploits this convergence by aligning frozen pretrained vision and language models with lightweight alignment layers, but typically relies on contrastive losses and millions of paired samples. In this work, we ask whether meaningful alignment can be achieved with substantially less supervision. We introduce a semi-supervised setting in which pretrained unimodal encoders are aligned using a small number of image-text pairs together with large amounts of unpaired data. To address this challenge, we propose SOTAlign, a two-stage framework that first recovers a coarse shared geometry from limited paired data using a linear teacher, and then refines the alignment on unpaired samples via an optimal-transport-based divergence that transfers relational structure without overconstraining the target space. SOTAlign effectively leverages unpaired images and text, learning robust joint embeddings across datasets and encoder pairs, and significantly outperforming supervised and semi-supervised baselines. Code is available at https://github.com/ExplainableML/SOTAlign.

视觉语言对齐半监督学习最优传输多模态

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。