用少量数据和计算资源,实现超分辨率图像生成的高效适配。
Ultra-Resolution Adaptation with Ease
- 利用教师模型生成合成数据提升训练效率,减少对真实数据依赖。
- 微调权重矩阵小部分成分,比低秩适配器更省参数且效果更好。
- 关闭分类器自由引导可显著提升模型在高分辨率下的生成质量。
近年来,文本到图像扩散模型取得了显著进展,但受限于数据与算力,高分辨率图像生成仍具挑战。本文从数据与参数效率出发,提出超分辨率适配指南URAE。理论与实证表明,部分教师模型生成的合成数据能显著加速收敛;在无合成数据时,微调权重矩阵的少量组件优于广泛使用的低秩适配器,兼具性能与效率。针对采用引导蒸馏的模型(如FLUX),发现将引导尺度设为1(即关闭分类器自由引导)对获得良好性能至关重要。大量实验验证,仅需3000样本与2000次迭代,URAE即可达到与FLUX1.1 [Pro] Ultra相当的2K生成水平,并在4K分辨率生成上创下新基准。代码已公开。
原文摘要 · Abstract (English)
Text-to-image diffusion models have achieved remarkable progress in recent years. However, training models for high-resolution image generation remains challenging, particularly when training data and computational resources are limited. In this paper, we explore this practical problem from two key perspectives: data and parameter efficiency, and propose a set of key guidelines for ultra-resolution adaptation termed \emph{URAE}. For data efficiency, we theoretically and empirically demonstrate that synthetic data generated by some teacher models can significantly promote training convergence. For parameter efficiency, we find that tuning minor components of the weight matrices outperforms widely-used low-rank adapters when synthetic data are unavailable, offering substantial performance gains while maintaining efficiency. Additionally, for models leveraging guidance distillation, such as FLUX, we show that disabling classifier-free guidance, \textit{i.e.}, setting the guidance scale to 1 during adaptation, is crucial for satisfactory performance. Extensive experiments validate that URAE achieves comparable 2K-generation performance to state-of-the-art closed-source models like FLUX1.1 [Pro] Ultra with only 3K samples and 2K iterations, while setting new benchmarks for 4K-resolution generation. Codes are available \href{https://github.com/Huage001/URAE}{here}.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。