让扩散模型自由生成任意分辨率图像,解决传统方法对固定分辨率的依赖。
FiTv2: Scalable and Improved Flexible Vision Transformer for Diffusion Model
- 将图像视为可变尺寸的令牌序列,实现训练与推理时的灵活分辨率适配。
- FiTv2收敛速度提升至FiT的2倍,支持高分辨率生成与跨分辨率扩展。
- 适合需要生成任意尺寸图像的研究者与开发者,尤其适用于多场景图像生成任务。
自然世界在分辨率上是无限自由的。现有扩散模型(如扩散变换器)在处理训练范围外的图像分辨率时面临挑战。为此,本文将图像视为具有动态尺寸的令牌序列,而非传统的固定分辨率网格,从而实现训练与推理中对不同长宽比的无缝适应,促进分辨率泛化并消除图像裁剪带来的偏差。基于此,提出专为无限制分辨率与长宽比图像生成设计的柔性视觉变压器(FiT)。进一步升级为FiTv2,引入查询-键向量归一化、AdaLN-LoRA模块、修正流调度器和Logit-Normal采样器,并优化网络结构,使收敛速度达到FiT的2倍。结合无需训练的外推技术,FiTv2在分辨率外推和多样化分辨率生成中表现出卓越适应性。此外,实验表明更大规模的FiTv2模型具有更优的计算效率。我们还提出一种高效的后训练策略,可将预训练模型快速适配高分辨率生成。大量实验证明,FiTv2在广泛分辨率下表现优异。代码与模型已开源:https://github.com/whlzy/FiT。
原文摘要 · Abstract (English)
\textit{Nature is infinitely resolution-free}. In the context of this reality, existing diffusion models, such as Diffusion Transformers, often face challenges when processing image resolutions outside of their trained domain. To address this limitation, we conceptualize images as sequences of tokens with dynamic sizes, rather than traditional methods that perceive images as fixed-resolution grids. This perspective enables a flexible training strategy that seamlessly accommodates various aspect ratios during both training and inference, thus promoting resolution generalization and eliminating biases introduced by image cropping. On this basis, we present the \textbf{Flexible Vision Transformer} (FiT), a transformer architecture specifically designed for generating images with \textit{unrestricted resolutions and aspect ratios}. We further upgrade the FiT to FiTv2 with several innovative designs, includingthe Query-Key vector normalization, the AdaLN-LoRA module, a rectified flow scheduler, and a Logit-Normal sampler. Enhanced by a meticulously adjusted network structure, FiTv2 exhibits $2\times$ convergence speed of FiT. When incorporating advanced training-free extrapolation techniques, FiTv2 demonstrates remarkable adaptability in both resolution extrapolation and diverse resolution generation. Additionally, our exploration of the scalability of the FiTv2 model reveals that larger models exhibit better computational efficiency. Furthermore, we introduce an efficient post-training strategy to adapt a pre-trained model for the high-resolution generation. Comprehensive experiments demonstrate the exceptional performance of FiTv2 across a broad range of resolutions. We have released all the codes and models at \url{https://github.com/whlzy/FiT} to promote the exploration of diffusion transformer models for arbitrary-resolution image generation.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。