Switti通过非因果设计与去引导优化，实现文本到图像生成7倍提速。
Switti: Designing Scale-Wise Transformers for Text-to-Image Synthesis
- 采用非因果结构替代传统自回归，提升推理速度并降低内存占用。
- 高分辨率阶段禁用分类器自由引导，使采样加速32%且细节更清晰。
- 在人类偏好和自动评估中超越主流自回归模型，媲美扩散模型。
本文提出Switti，一种用于文本到图像生成的尺度感知变压器。首先将现有下一尺度预测自回归（AR）架构适配至T2I任务，并解决其训练不稳定性问题。接着，我们指出尺度感知变压器无需因果性，提出非因果版本，实现约21%的采样加速与更低内存消耗，同时生成质量略有提升。此外，发现高分辨率尺度上的分类器自由引导常无必要，甚至会损害性能；禁用该机制后，采样速度再提升约32%，细粒度细节生成效果改善。大量人工偏好测试与自动评估表明，Switti优于现有T2I AR模型，且在性能上可比肩顶尖扩散模型，最快可达7倍加速。
原文摘要 · Abstract (English)
This work presents Switti, a scale-wise transformer for text-to-image generation. We start by adapting an existing next-scale prediction autoregressive (AR) architecture to T2I generation, investigating and mitigating training stability issues in the process. Next, we argue that scale-wise transformers do not require causality and propose a non-causal counterpart facilitating ~21% faster sampling and lower memory usage while also achieving slightly better generation quality. Furthermore, we reveal that classifier-free guidance at high-resolution scales is often unnecessary and can even degrade performance. By disabling guidance at these scales, we achieve an additional sampling acceleration of ~32% and improve the generation of fine-grained details. Extensive human preference studies and automated evaluations show that Switti outperforms existing T2I AR models and competes with state-of-the-art T2I diffusion models while being up to 7x faster.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。