统一视觉语言理解生成检索,用隐空间对齐提升多模态通用性
OmniBridge: Unified Multimodal Understanding, Generation, and Retrieval via Latent Space Alignment
- 以语言模型为中心,通过轻量级双向隐空间对齐实现多任务统一
- 在多个基准上达到领先或竞争性表现,验证了隐空间对齐的有效性
- 适合需要跨模态统一建模的研究者与开发者使用
近年来多模态大语言模型在理解、生成和检索任务上取得显著进展,但现有方法常孤立处理各任务或需从头训练模型,导致计算成本高且模态泛化能力有限。本文提出OmniBridge,一种统一且模块化的多模态框架,支持视觉-语言理解、生成与检索的统一架构。该框架采用语言中心设计,复用预训练大语言模型,并引入轻量级双向隐空间对齐模块。为缓解任务干扰,提出两阶段解耦训练策略:监督微调与隐空间对齐,用于对齐大语言模型行为与多模态推理;以及语义引导扩散训练,通过可学习查询嵌入对齐跨模态隐空间。大量实验表明,OmniBridge在各项任务上均达到竞争力或最先进性能。结果证实,隐空间对齐在共享表示空间中统一多模态建模的有效性。代码与模型已开源于https://github.com/xiao-xt/OmniBridge。
原文摘要 · Abstract (English)
Recent advances in multimodal large language models (LLMs) have led to significant progress in understanding, generation, and retrieval tasks. However, current solutions often treat these tasks in isolation or require training LLMs from scratch, resulting in high computational costs and limited generalization across modalities. In this work, we present OmniBridge, a unified and modular multimodal framework that supports vision-language understanding, generation, and retrieval within a unified architecture. OmniBridge adopts a language-centric design that reuses pretrained LLMs and introduces a lightweight bidirectional latent alignment module. To address the challenge of task interference, we propose a two-stage decoupled training strategy: supervised fine-tuning and latent space alignment for aligning LLM behavior with multimodal reasoning, and semantic-guided diffusion training to align cross-modal latent spaces via learnable query embeddings. Extensive experiments across a wide range of benchmarks demonstrate that OmniBridge achieves competitive or state-of-the-art performance in all three tasks. Moreover, our results highlight the effectiveness of latent space alignment for unifying multimodal modeling under a shared representation space. Code and models are released at https://github.com/xiao-xt/OmniBridge.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。