将扩散模型与自回归生成结合，实现跨模态高效生成。
Causal Diffusion Transformers for Generative Modeling
- 通过序列因子分解，统一自回归与扩散生成框架。
- 在ImageNet上达到当前最优生成效果，支持任意长度推理。
- 可零样本完成图像编辑与图文联合生成，适合多模态研究者。
我们提出因果扩散（Causal Diffusion），作为扩散模型的自回归（AR）对应物。它是一种面向离散与连续模态的下一令牌预测框架，兼容现有模型如LLaMA和GPT。尽管已有工作尝试融合扩散与自回归模型，我们发现对扩散模型引入序列因子化可显著提升性能，并实现自回归与扩散生成模式间的平滑切换。为此，我们提出CausalFusion——一种仅解码器的Transformer，同时对序列令牌与扩散噪声层级进行双因子分解，在ImageNet生成基准上取得当前最优结果，且具备自回归生成任意长度令牌的能力，适用于上下文推理。进一步通过联合图像生成与描述模型展示其多模态能力，并验证其零样本上下文图像操作能力。本工作为离散与连续数据上的多模态训练提供了新视角。
原文摘要 · Abstract (English)
We introduce Causal Diffusion as the autoregressive (AR) counterpart of Diffusion models. It is a next-token(s) forecasting framework that is friendly to both discrete and continuous modalities and compatible with existing next-token prediction models like LLaMA and GPT. While recent works attempt to combine diffusion with AR models, we show that introducing sequential factorization to a diffusion model can substantially improve its performance and enables a smooth transition between AR and diffusion generation modes. Hence, we propose CausalFusion - a decoder-only transformer that dual-factorizes data across sequential tokens and diffusion noise levels, leading to state-of-the-art results on the ImageNet generation benchmark while also enjoying the AR advantage of generating an arbitrary number of tokens for in-context reasoning. We further demonstrate CausalFusion's multimodal capabilities through a joint image generation and captioning model, and showcase CausalFusion's ability for zero-shot in-context image manipulations. We hope that this work could provide the community with a fresh perspective on training multimodal models over discrete and continuous data.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。