用二维自回归提升图像生成质量，突破向量量化瓶颈。
A Spark of Vision-Language Intelligence: 2-Dimensional Autoregressive Transformer for Efficient Finegrained Image Generation
- 引入模型深度方向的二维自回归机制，缓解信息丢失。
- 相同参数量下生成质量优于传统1D与RQ-Transformer。
- 无需文本标签即可生成图文混合图像，展现视觉语言理解雏形。
本文针对向量量化自回归图像生成中的信息损失瓶颈，提出新型架构2维自回归（DnD）Transformer。该模型在序列长度方向外新增模型深度方向作为新的自回归维度，从而预测更多图像码本。相较于传统1D自回归及类似2D分解的RQ-Transformer，DnD-Transformer为端到端模型，在相同骨干模型规模与序列长度下生成更高品质图像，开辟了自回归图像生成的新优化路径。实验还发现，DnD-Transformer的潜力不仅限于自然图像生成，还能在无监督条件下生成富含文本与图形元素的图像，展现出对多模态内容的理解能力。这一现象在主流扩散模型等视觉生成模型中尚未实现，表明仅训练于图像数据时已具备视觉-语言智能的初步迹象。代码、数据集与模型已在https://github.com/chenllliang/DnD-Transformer公开。
原文摘要 · Abstract (English)
This work tackles the information loss bottleneck of vector-quantization (VQ) autoregressive image generation by introducing a novel model architecture called the 2-Dimensional Autoregression (DnD) Transformer. The DnD-Transformer predicts more codes for an image by introducing a new autoregression direction, \textit{model depth}, along with the sequence length direction. Compared to traditional 1D autoregression and previous work utilizing similar 2D image decomposition such as RQ-Transformer, the DnD-Transformer is an end-to-end model that can generate higher quality images with the same backbone model size and sequence length, opening a new optimization perspective for autoregressive image generation. Furthermore, our experiments reveal that the DnD-Transformer's potential extends beyond generating natural images. It can even generate images with rich text and graphical elements in a self-supervised manner, demonstrating an understanding of these combined modalities. This has not been previously demonstrated for popular vision generative models such as diffusion models, showing a spark of vision-language intelligence when trained solely on images. Code, datasets and models are open at https://github.com/chenllliang/DnD-Transformer.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。