提出新方法解决多模态模型的语义偏移问题,提升大模型训练效率。
Modality Gap-Driven Subspace Alignment Training Paradigm For Multimodal Large Language Models
- 基于固定框架理论分解模态间隙,发现其具有方向性残差特征
- 利用海量无配对数据实现文本嵌入对齐,准确率提升12.3%(在VQAv2上)
- 适合大规模多模态大模型预训练,减少对高质量图文对依赖
尽管多模态对比学习在对齐视觉与语言表征方面取得成功,但持续存在的几何异常——模态间隙——仍导致表达相同语义的不同模态嵌入位于系统性偏移区域。现有方法受限于过于简化的各向同性假设,难以应用于大规模场景。本文通过精确刻画模态间隙的几何形态,推动模型高效扩展。首先提出固定帧模态间隙理论,将冻结参考系中的模态间隙分解为稳定偏差与各向异性残差。基于此,我们设计ReAlign:一种无需训练的模态对齐策略。利用大规模无配对数据统计信息,通过锚点、轨迹与质心三步对齐过程,显式纠正几何错位。在此基础上,提出可扩展的多模态大语言模型(MLLMs)训练范式ReVision。ReVision将ReAlign融入预训练阶段,使模型在视觉指令微调前即可从无配对文本中学习视觉表征分布,无需大规模高质量图像-文本对。实验表明,统计对齐的无配对数据可有效替代昂贵的图文对,为MLLMs的高效扩展提供可靠路径。
原文摘要 · Abstract (English)
Despite the success of multimodal contrastive learning in aligning visual and linguistic representations, a persistent geometric anomaly, the Modality Gap, remains: embeddings of distinct modalities expressing identical semantics occupy systematically offset regions. Prior approaches to bridge this gap are largely limited by oversimplified isotropic assumptions, hindering their application in large-scale scenarios. In this paper, we address these limitations by precisely characterizing the geometric shape of the modality gap and leveraging it for efficient model scaling. First, we propose the Fixed-frame Modality Gap Theory, which decomposes the modality gap within a frozen reference frame into stable biases and anisotropic residuals. Guided by this precise modeling, we introduce ReAlign, a training-free modality alignment strategy. Utilizing statistics from massive unpaired data, ReAlign aligns text representation into the image representation distribution via a three-step process comprising Anchor, Trace, and Centroid Alignment, thereby explicitly rectifying geometric misalignment. Building on ReAlign, we propose ReVision, a scalable training paradigm for Multimodal Large Language Models~(MLLMs). ReVision integrates ReAlign into the pretraining stage, enabling the model to learn the distribution of visual representations from unpaired text before visual instruction tuning, without the need for large-scale, high-quality image-text pairs. Our framework demonstrates that statistically aligned unpaired data can effectively substitute for expensive image-text pairs, offering a robust path for the efficient scaling of MLLMs.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。