用离散+连续双阶段生成图像,兼顾质量与效率。
D2C: Unlocking the Potential of Continuous Autoregressive Image Generation with Discrete Tokens
- 先用小模型生成离散粗粒度特征,再用连续特征细化细节。
- ImageNet-256上优于多个离散与连续生成模型。
- 适合追求高质量且高效图像生成的研究者。
在图像生成领域,基于潜在空间的生成模型占据主导地位,但高度依赖图像分词器。自回归模型虽具可扩展性和灵活性,采用离散分词器,却面临生成质量差的问题;扩散模型利用连续分词器,生成质量好,但效率低、结构复杂。现有混合模型多用于弥补信息损失和简化扩散学习过程。离散与连续分词器融合的潜力尚未探索。本文提出D2C,一种两阶段方法以提升生成能力:第一阶段通过小型离散生成器采样表示粗粒度图像特征的离散令牌;第二阶段在离散令牌序列条件下学习表示细粒度特征的连续令牌。同时设计两类融合模块实现无缝交互。在ImageNet-256基准上,实验表明该模型在类别条件图像生成任务中表现优于多个连续与离散生成模型。
原文摘要 · Abstract (English)
In the domain of image generation, latent-based generative models occupy a dominant status; however, these models rely heavily on image tokenizer. To meet modeling requirements, autoregressive models possessing the characteristics of scalability and flexibility embrace a discrete-valued tokenizer, but face the challenge of poor image generation quality. In contrast, diffusion models take advantage of the continuous-valued tokenizer to achieve better generation quality but are subject to low efficiency and complexity. The existing hybrid models are mainly to compensate for information loss and simplify the diffusion learning process. The potential of merging discrete-valued and continuous-valued tokens in the field of image generation has not yet been explored. In this paper, we propose D2C, a novel two-stage method to enhance model generation capacity. In the first stage, the discrete-valued tokens representing coarse-grained image features are sampled by employing a small discrete-valued generator. Then in the second stage, the continuous-valued tokens representing fine-grained image features are learned conditioned on the discrete token sequence. In addition, we design two kinds of fusion modules for seamless interaction. On the ImageNet-256 benchmark, extensive experiment results validate that our model achieves superior performance compared with several continuous-valued and discrete-valued generative models on the class-conditional image generation tasks.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。