通过显式对齐编码与解码路径,提升多模态模型生成与理解的一致性。
LatentUMM: Dual Latent Alignment for Unified Multimodal Models

- 双潜空间对齐:在模态和容量层面强制编码与解码路径一致
- 跨模态一致性显著提升,尤其在跨模态转换中减少语义漂移
- 适用于需高一致性的多模态生成与理解任务,如视觉-语言模型
统一多模态模型(UMMs)通过学习共享潜空间在理解和生成任务上表现优异,但二者常存在功能不一致。我们发现该问题并非缺乏共享表征,而是进入和离开潜空间的变换缺乏显式对齐。这导致生成与重编码路径不一致,引发模态转换时的语义漂移。本文提出LatentUMM框架,构建增强的共享潜空间以显式对齐这些变换并提升跨模态一致性。该框架分两阶段:第一阶段,双潜空间对齐在模态与容量层面实现一致性——跨模态对齐使用更强嵌入模型施加结构化跨模态语义,双重容量对齐确保生成与重编码下的双向一致性;第二阶段,潜空间动态稳定通过随机潜空间回溯与偏好优化,优选更保持语义一致性的轨迹。实验表明,LatentUMM在多种架构下均显著提升多模态一致性。代码已公开于:https://github.com/AIFrontierLab/TorchUMM/tree/main/src/umm/post_training/LatentUMM。
原文摘要 · Abstract (English)
Unified multimodal models (UMMs) achieve strong performance in both understanding and generation by learning a shared latent space, yet they often exhibit functional inconsistency between these two capabilities. We observe that this issue does not stem from a lack of shared representations, but from the absence of explicit alignment between the transformations that map into and out of the latent space. As a result, generation and re-encoding can follow inconsistent trajectories, leading to semantic drift under modality transitions. In this work, we propose LatentUMM, a framework that constructs an enhanced shared latent space to explicitly align these transformations and improve cross-modal consistency. LatentUMM consists of two stages. First, dual latent alignment enforces consistency at both the modality and capacity levels: cross-modal alignment uses a stronger embedding model to impose structured cross-modal semantics, while dual capacity alignment enforces bidirectional consistency under generation and re-encoding. Second, latent dynamics stabilization improves robustness via stochastic latent rollouts and preference optimization, favoring trajectories that better preserve semantic consistency. Experiments show that LatentUMM consistently improves multimodal consistency across diverse architectures. Code is available at: https://github.com/AIFrontierLab/TorchUMM/tree/main/src/umm/post_training/LatentUMM.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。