LEDiT无需位置编码即可实现图像分辨率4倍扩展。
LEDiT: Your Length-Extrapolatable Diffusion Transformer without Positional Encoding
- 用因果注意力隐式编码位置信息，避免显式位置编码外推
- 在256x256训练下实现512x512生成，图像质量优于现有方法
- 适合需要高分辨率生成且不依赖位置编码的场景
扩散变压器（DiT）在生成高于训练分辨率的图像时表现不佳，主要原因是显式位置编码（如RoPE）需外推至未见位置，导致性能下降。本文提出长度可外推的扩散变压器（LEDiT），无需显式位置编码，从而避免编码外推问题。LEDiT的核心创新在于使用因果注意力，我们证明其能隐式编码全局位置信息，并有助于外推。此外，引入局部增强模块以捕捉细粒度局部信息，弥补因果注意力提供的粗粒度全局位置信息。在条件生成和文本到图像生成任务上的实验表明，LEDiT支持最高4倍分辨率扩展（如从256x256到512x512），图像质量优于当前最先进的外推方法。我们认为LEDiT标志着对标准RoPE方法的突破，为长度外推提供了新思路。
原文摘要 · Abstract (English)
Diffusion transformers (DiTs) struggle to generate images at resolutions higher than their training resolutions. The primary obstacle is that the explicit positional encodings(PE), such as RoPE, need extrapolating to unseen positions which degrades performance when the inference resolution differs from training. In this paper, We propose a Length-Extrapolatable Diffusion Transformer~(LEDiT) to overcome this limitation. LEDiT needs no explicit PEs, thereby avoiding PE extrapolation. The key innovation of LEDiT lies in the use of causal attention. We demonstrate that causal attention can implicitly encode global positional information and show that such information facilitates extrapolation. We further introduce a locality enhancement module, which captures fine-grained local information to complement the global coarse-grained position information encoded by causal attention. Experimental results on both conditional and text-to-image generation tasks demonstrate that LEDiT supports up to 4x resolution scaling (e.g., from 256x256 to 512x512), achieving better image quality compared to the state-of-the-art length extrapolation methods. We believe that LEDiT marks a departure from the standard RoPE-based methods and offers a promising insight into length extrapolation. Project page: https://shenzhang2145.github.io/ledit/
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。