提出随机位置编码,让扩散模型跨分辨率生成更稳定。
Boosting Resolution Generalization of Diffusion Transformers with Randomized Positional Encodings
- 用随机采样位置编码替代固定距离编码,关注图像块顺序
- 在256²训练时生成384²和512²图像,性能领先
- 适合需要跨分辨率生成、低分辨率训练加速的场景
图像生成中的分辨率泛化能力可实现以较低训练分辨率开销生成更高分辨率图像。然而,扩散变压器面临的关键障碍是推理时与训练时位置编码不匹配。现有插值、外推或混合策略未能完全解决此问题。本文提出一种新型二维随机位置编码RPE-2D,优先考虑图像块的相对顺序而非绝对距离,实现无需多分辨率训练即可无缝支持高/低分辨率生成。具体地,RPE-2D在训练时沿水平和垂直轴独立从扩展范围采样位置,确保推理时编码落在训练分布内,从而提升分辨率泛化能力。我们进一步引入随机缩放裁剪增强以强化顺序建模,并加入微条件控制标注裁剪模式。在ImageNet数据集上,RPE-2D在256²训练下评估384²和512²,以及512²训练下评估768²和1024²时均达到当前最优性能;同时在低分辨率生成、多阶段训练加速和多分辨率继承方面表现优异。
原文摘要 · Abstract (English)
Resolution generalization in image generation tasks enables the production of higher-resolution images with lower training resolution overhead. However, a key obstacle for diffusion transformers in addressing this problem is the mismatch between positional encodings seen at inference and those used during training. Existing strategies such as positional encodings interpolation, extrapolation, or hybrids, do not fully resolve this mismatch. In this paper, we propose a novel two-dimensional randomized positional encodings, namely RPE-2D, that prioritizes the order of image patches rather than their absolute distances, enabling seamless high- and low-resolution generation without training on multiple resolutions. Concretely, RPE-2D independently samples positions along the horizontal and vertical axes over an expanded range during training, ensuring that the encodings used at inference lie within the training distribution and thereby improving resolution generalization. We further introduce a simple random resize-and-crop augmentation to strengthen order modeling and add micro-conditioning to indicate the applied cropping pattern. On the ImageNet dataset, RPE-2D achieves state-of-the-art resolution generalization performance, outperforming competitive methods when trained at $256^2$ and evaluated at $384^2$ and $512^2$, and when trained at $512^2$ and evaluated at $768^2$ and $1024^2$. RPE-2D also exhibits outstanding capabilities in low-resolution image generation, multi-stage training acceleration, and multi-resolution inheritance.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。