统一视觉理解与生成任务,用分离解码提升效果与可扩展性
UniDDT: Unifying Multimodal Understanding and Generation with Decoupled Diffusion Transformer

- 采用噪声视觉变压器编码器+大语言模型统一语义表示
- 生成任务得分0.87(GenEval),理解任务1699.5(MME)
- 双数据结构设计增强图文任务内在关联,适合多模态研究者
统一多模态模型(UMMs)已成为通用多模态智能的关键方向,将理解与生成整合于单一框架。然而现有模型存在三大挑战:(1) 视觉理解与生成任务间学习冲突,导致两者建模均不理想;(2) 理解与生成的视觉空间不同,限制可扩展性;(3) 过度依赖特定任务数据,忽视文本-图像理解与生成的双重性。为此,我们提出UniDDT,利用噪声视觉变压器(Noisy ViT)编码器结合大语言模型(LLM),统一视觉生成与理解的语义编码,并通过独立扩散解码器将扩散解码与文本解码分离。该设计使模型能以潜在空间作为统一视觉表征,实现理解与生成任务的无缝兼容,平衡生成可扩展性与理解语义表达力。此外,基于同一图像-文本对构建双数据结构,促进生成与理解数据间的相互依赖,挖掘其内在双重性。大量实验表明,UniDDT在多模态理解与生成任务中实现了有效统一,显著提升语义一致性与可扩展性。在视觉生成任务上,获得0.87的GenEval得分和86.9的DPG总体得分;在多模态理解任务上,于MME基准达1699.5分,于SEEDbench得76.5总体分。
原文摘要 · Abstract (English)
Unified Multimodal Models (UMMs) have emerged as a critical direction for general-purpose multimodal intelligence, integrating understanding and generation into a single framework. However, existing UMMs face prominent challenges: (1) the inherent learning conflicts between visual understanding and generation tasks, leading to suboptimal modeling in both tasks; (2) different understanding and generation visual spaces impeding scalability; (3) over-reliance on task-specific data that neglects the duality of text-image understanding and generation. To address these challenges, we propose UniDDT, which leverages a Noisy ViT encoder along with an LLM to unify semantic encoding for visual generation and understanding tasks, while employing a separate diffusion decoder to decouple diffusion decoding from text decoding. With this Noisy ViT encoder, UniDDT is able to leverage the latent space as a unified visual representation, enabling seamless compatibility between understanding and generation tasks. Thus, the scalability within the generation tasks and the semantic expressiveness within understanding tasks can be balanced. Also, we construct dual data structures from the same image-text pairs, fostering interdependence between the generation and understanding data to exploit their inherent duality. Extensive experiments demonstrate that UniDDT achieves effective unification of multimodal understanding and generation with enhanced semantic consistency and scalability. For visual generation tasks, our UniDDT achieves 0.87 GenEval score and 86.9 DPG overall score. For multimodal understanding tasks, our UniDDT achieves 1699.5 score on MME benchmark and 76.5 overall score on SEEDbench.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。