用同一Transformer同时实现文本与图像生成，性能不降反升。
MonoFormer: One Transformer for Both Diffusion and Autoregression
- 共享一个Transformer，通过注意力掩码区分自回归与扩散
- 图像生成质量媲美顶尖方法，同时保留文本生成能力
- 适合想统一多模态生成框架的研究者
现有多数多模态方法为自回归式离散文本生成和扩散式连续视觉生成分别设计骨干网络，或通过离散化视觉数据统一使用自回归。本文提出一种简单思路：共享一个Transformer处理自回归与扩散。可行性基于两点：(i) Transformer已在扩散模型中成功用于视觉生成；(ii) 自回归与扩散的Transformer训练高度相似，差异仅在于扩散使用双向注意力掩码，自回归使用因果注意力掩码。实验表明，该方法在图像生成性能上达到当前最先进水平，同时保持文本生成能力。项目开源地址：https://monoformer.github.io/。
原文摘要 · Abstract (English)
Most existing multimodality methods use separate backbones for autoregression-based discrete text generation and diffusion-based continuous visual generation, or the same backbone by discretizing the visual data to use autoregression for both text and visual generation. In this paper, we propose to study a simple idea: share one transformer for both autoregression and diffusion. The feasibility comes from two main aspects: (i) Transformer is successfully applied to diffusion for visual generation, and (ii) transformer training for autoregression and diffusion is very similar, and the difference merely lies in that diffusion uses bidirectional attention mask and autoregression uses causal attention mask. Experimental results show that our approach achieves comparable image generation performance to current state-of-the-art methods as well as maintains the text generation capability. The project is publicly available at https://monoformer.github.io/.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。