将图像细节与语义分离,实现图文理解生成统一建模
Cheers: Decoupling Patch Details from Semantic Representations Enables Unified Multimodal Comprehension and Generation
- 通过解耦图像块级细节与语义表征,稳定理解并提升生成质量
- 在GenEval和MMBench上优于Tar-1.5B,训练成本仅为20%
- 支持4倍令牌压缩,高效实现高分辨率图像生成
多模态建模的前沿方向是统一视觉理解与生成任务。然而,两项任务对解码方式和视觉表征的需求不匹配,难以在共享特征空间中联合优化。本文提出Cheers模型,将图像块级细节与语义表征解耦,通过门控细节残差稳定语义表示并提升生成保真度。其包含三个核心组件:(i) 统一视觉分词器,将图像潜在状态编码为语义令牌以高效适配大语言模型;(ii) 基于LLM的Transformer,统一文本生成的自回归解码与图像生成的扩散解码;(iii) 级联流匹配头,先解码视觉语义,再注入来自分词器的语义门控细节残差以细化高频内容。在主流基准测试中,Cheers在视觉理解与生成方面均达到或超越先进多模态模型水平。同时实现4倍令牌压缩,显著提升高分辨率图像编码与生成效率。值得注意的是,Cheers在GenEval和MMBench上表现优于Tar-1.5B,且仅需其20%训练成本,验证了高效统一多模态建模的可行性。代码与数据将公开。
原文摘要 · Abstract (English)
A recent cutting-edge topic in multimodal modeling is to unify visual comprehension and generation within a single model. However, the two tasks demand mismatched decoding regimes and visual representations, making it non-trivial to jointly optimize within a shared feature space. In this work, we present Cheers, a unified multimodal model that decouples patch-level details from semantic representations, thereby stabilizing semantics for multimodal understanding and improving fidelity for image generation via gated detail residuals. Cheers includes three key components: (i) a unified vision tokenizer that encodes and compresses image latent states into semantic tokens for efficient LLM conditioning, (ii) an LLM-based Transformer that unifies autoregressive decoding for text generation and diffusion decoding for image generation, and (iii) a cascaded flow matching head that decodes visual semantics first and then injects semantically gated detail residuals from the vision tokenizer to refine high-frequency content. Experiments on popular benchmarks demonstrate that Cheers matches or surpasses advanced UMMs in both visual understanding and generation. Cheers also achieves 4x token compression, enabling more efficient high-resolution image encoding and generation. Notably, Cheers outperforms the Tar-1.5B on the popular benchmarks GenEval and MMBench, while requiring only 20% of the training cost, indicating effective and efficient (i.e., 4x token compression) unified multimodal modeling. We will release all code and data for future research.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。