arXiv:2502.05415cs.CVcs.AI2025-02ACL被引 1

统一模型实现图文生成与理解,效率远超现有方法。

UniCMs: A Unified Consistency Model For Efficient Multimodal Generation and Understanding

  • 用统一的去噪轨迹训练图文模型,文本采用自回归解码路径。
  • 图文生成速度提升8倍,图像质量优于SD3,文本生成更快更准。
  • 适合需要高效多模态生成的开发者与研究者使用。

一致性模型在图像和文本生成中展现出高效潜力,这促使我们思考能否构建一个统一的一致性模型,实现高效的多模态生成(如文生图)与理解(如图生文)。直观上可通过将一致性蒸馏(CD)应用于现有统一多模态模型来实现,但关键挑战在于为图像和文本生成建立统一的去噪视角,以确保一致性映射。为此,我们在表示层面为两种模态均采用离散标记,以更好保留语言建模能力。特别地,不同于基于离散扩散的语言建模原理,我们采用自回归语言模型的并行解码轨迹定义文本去噪路径,从而利用其在通用文本生成中的优异表现。图像标记的去噪轨迹遵循标准离散扩散。我们通过联合多模态轨迹,以统一目标同时训练统一一致性模型(UniCMs),并引入轨迹分段策略以提升训练收敛性。实验表明,在文生图任务中,UniCMs 在 GenEval、Image Reward 与 CLIP Score 上优于 SD3,采样时间仅为其约 1/8;在图生文任务中,优于 Show-o,长序列生成速度提升 1.5 倍。代码已开源。

原文摘要 · Abstract (English)

Consistency models (CMs) have shown promise in the efficient generation of both image and text. This raises the natural question of whether we can learn a unified CM for efficient multimodal generation (e.g., text-to-image) and understanding (e.g., image-to-text). Intuitively, such a model could be acquired by applying the consistency distillation (CD) to existing unified multimodal models. However, the key challenge is establishing a unified denoising perspective for both image and text generation, which is essential for establishing the consistency mapping. To tackle this, at the representation level, we advocate for discrete tokens for both modalities to best preserve language modeling capabilities. Critically, instead of defining the text denoising trajectory via recent discrete diffusion language modeling principles, we specify it using the parallel decoding trace of an autoregressive language model, benefiting from the latter's superior performance in general text generation tasks. The denoising trajectory of image tokens adheres to standard discrete diffusion. We train our unified consistency models (UniCMs) on these combined multimodal trajectories simultaneously with a unified objective. We introduce a trajectory segmentation strategy to further improve the training convergence. Empirically, in text-to-image generation, UniCMs outperform SD3 on GenEval, Image Reward, and CLIP Score metrics, while requiring only approximately ${1}/{8}$ of the sampling time. Meanwhile, in image-to-text generation, UniCMs surpass Show-o on the MMMU benchmark while being $1.5 \times$ faster at long-sequence generating speed. The code is available at https://github.com/zhijie-group/UniCMs.

多模态一致性模型生成效率图文生成

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。