针对高分辨率生成模型的推理设计优化，提升效率与质量。
Bag of Design Choices for Inference of High-Resolution Masked Generative Transformer
- 提出专为掩码生成Transformer设计的推理优化方法
- 在Meissonic-1024x1024上实现约70%胜率优于原始采样
- 适合关注生成模型推理加速与高质量图像生成的研究者
文本到图像扩散模型发展迅速，但其与自回归模型之间的差异阻碍了视觉与语言统一生成的目标。最近，掩码生成Transformer（MGT）通过预测随机掩码的图像标记，成为连接扩散模型与自回归模型的有前景桥梁，兼具扩散模型的高效性与自回归模型的离散性。然而，关于MGT推理的全面分析几乎空白。本文提出并重构了一系列面向MGT的增强推理技术，进行了详细性能分析，并探索了基于扩散模型的方法以加速MGT采样过程。在当前SOTA的MGT模型如MaskGIT和Meissonic上进行广泛实验，得出具体有效的设计选择，且可融合使用以进一步提升性能。例如，在HPS v2基准下，使用Meissonic-1024x1024时，增强推理相比原始采样获得约70%的胜率。
原文摘要 · Abstract (English)
Text-to-image diffusion models (DMs) develop at an unprecedented pace, supported by thorough theoretical exploration and empirical analysis. Unfortunately, the discrepancy between DMs and autoregressive models (ARMs) complicates the path toward achieving the goal of unified vision and language generation. Recently, the masked generative Transformer (MGT) serves as a promising intermediary between DM and ARM by predicting randomly masked image tokens (i.e., masked image modeling), combining the efficiency of DM with the discrete token nature of ARM. However, we find that the comprehensive analyses regarding the inference for MGT are virtually non-existent, and thus we aim to present positive design choices to fill this gap. We propose and redesign a set of enhanced inference techniques tailored for MGT, providing a detailed analysis of their performance. Additionally, we explore several DM-based approaches aimed at accelerating the sampling process on MGT. Extensive experiments and empirical analyses on the recent SOTA MGT, such as MaskGIT and Meissonic lead to concrete and effective design choices, and these design choices can be merged to achieve further performance gains. For instance, in terms of enhanced inference, we achieve winning rates of approximately 70% compared to vanilla sampling on HPS v2 with Meissonic-1024x1024.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。