首个统一图文生成的离散扩散模型，支持高效可控的文本图像联合生成。
Unified Multimodal Discrete Diffusion
- 采用离散扩散机制统一建模文本与图像，突破传统自回归顺序生成限制。
- 在图像生成与图文编辑任务中，性能优于自回归模型，推理效率更高。
- 适合需要灵活控制生成质量与速度的研究者及应用开发者。
多模态生成模型通常依赖自回归（AR）方法，按从左到右或从上到下的顺序处理文本、图像、视频和音频等多模态数据，完成图像描述、问答和图像生成等任务。本文探索将离散扩散模型作为文本与图像联合领域的统一生成范式，基于其在文本生成中的近期成功。离散扩散模型相比自回归模型具有诸多优势：可更好控制生成样本的质量与多样性，支持跨文本与图像域的联合修复（inpainting），并通过引导实现更强的生成可控性。我们提出首个统一多模态离散扩散模型（UniDisc），可在多种下游任务中联合理解与生成文本和图像。与多模态自回归模型对比，通过缩放分析表明，UniDisc在性能与推理计算成本方面均更优，且具备更强的可控性、可编辑性、修复能力，以及灵活调节推理时间与生成质量的权衡能力。代码与可视化见 https://unidisc.github.io。
原文摘要 · Abstract (English)
Multimodal generative models that can understand and generate across multiple modalities are dominated by autoregressive (AR) approaches, which process tokens sequentially from left to right, or top to bottom. These models jointly handle images, text, video, and audio for various tasks such as image captioning, question answering, and image generation. In this work, we explore discrete diffusion models as a unified generative formulation in the joint text and image domain, building upon their recent success in text generation. Discrete diffusion models offer several advantages over AR models, including improved control over quality versus diversity of generated samples, the ability to perform joint multimodal inpainting (across both text and image domains), and greater controllability in generation through guidance. Leveraging these benefits, we present the first Unified Multimodal Discrete Diffusion (UniDisc) model which is capable of jointly understanding and generating text and images for a variety of downstream tasks. We compare UniDisc to multimodal AR models, performing a scaling analysis and demonstrating that UniDisc outperforms them in terms of both performance and inference-time compute, enhanced controllability, editability, inpainting, and flexible trade-off between inference time and generation quality. Code and additional visualizations are available at https://unidisc.github.io.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。