解决扩散模型中多分辨率注意力的错位问题，提升生成质量
Phase-Aligned RoPE for Mixed-Resolution Diffusion Transformer
- 提出相位对齐机制，确保不同分辨率间位置编码一致
- 在图像和视频生成任务中显著提升视觉质量与效率
- 无需训练，适合现有扩散模型直接部署
旋转位置编码（RoPE）广泛用于扩散变压器（DiTs）中编码空间关系，但其在多分辨率标记中的表现仍不明确。一种自然方法是将不同分辨率的标记位置统一缩放后进行注意力计算，但我们发现这种方法无效。分析表明，使用RoPE时，注意力相似度是标记距离的高度结构化且周期性函数，跨分辨率缩放会改变标记对在该周期函数中的位置，导致错误的注意力分数。为此，我们提出无需训练的相位对齐混合分辨率注意力（PMA），通过调整RoPE位置映射，为每个查询-键对保持一致的位置尺度，确保相对距离始终在统一参考尺度下评估。为进一步增强分辨率过渡处的局部一致性，引入轻量级边界精修模块，软性交换相邻尺度特征。在图像和视频扩散模型上的实验验证了分析，并展现出稳定的视觉保真度与计算效率提升。
原文摘要 · Abstract (English)
Rotary positional embeddings (RoPE) are widely used in diffusion transformers (DiTs) to encode spatial relationships, yet their behavior with mixed-resolution tokens remains underexplored. A natural approach is to rescale token positions from different resolutions into a unified coordinate system before attention, but we show this fails. Our analysis shows that with RoPE, the attention similarity score is a highly structured and periodic function of token distance, so rescaling distances across resolutions moves token pairs to different regions of this periodic function, leading to incorrect attention scores. Motivated by this, we introduce Phase-Aligned Mixed-Resolution Attention (PMA), a training-free mechanism that stabilizes mixed-resolution attention. PMA modifies the RoPE position mapping to enforce a consistent positional scale for every query-key pair, ensuring that relative distances are evaluated under a single reference scale. To further improve local coherence near resolution transitions, we incorporate a lightweight boundary refinement module that softly exchanges features across adjacent scales. Experiments on image and video diffusion models validate our analysis and demonstrate consistent improvements in visual fidelity and computational efficiency.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。