通过调控旋转位置编码频率,解决多模态生成中的参考内容复制问题。
Untwisting RoPE: Frequency Control for Shared Attention in DiTs
- 分析RoPE的频段特性,发现高频成分主导注意力
- 高频成分导致生成时复制参考图内容而非仅提取风格
- 可调频的注意力机制实现风格迁移不复制内容
位置编码对基于Transformer的生成模型至关重要,但在多模态和共享注意力场景下的行为尚未完全理解。本文对旋转位置编码(RoPE)进行系统分析,发现其天然分解为具有不同位置敏感性的频率成分。我们证明,这一频率结构解释了共享注意力机制中出现的参考内容复制现象:当目标图像生成时同时关注参考图像的标记,模型会复制参考内容而非仅提取其风格特征。分析显示,RoPE的高频成分主导注意力计算,迫使查询主要关注空间对齐的参考标记,从而引发意外的复制行为。基于此,我们提出一种选择性调节RoPE频段的方法,使注意力反映语义相似性而非严格的位置对齐。该方法应用于现代基于Transformer的扩散架构(所有标记共享注意力),恢复了稳定且有意义的共享注意力。结果可有效控制风格迁移与内容复制的程度,实现风格一致但不复制参考内容的生成过程。
原文摘要 · Abstract (English)
Positional encodings are essential to transformer-based generative models, yet their behavior in multimodal and attention-sharing settings is not fully understood. In this work, we present a principled analysis of Rotary Positional Embeddings (RoPE), showing that RoPE naturally decomposes into frequency components with distinct positional sensitivities. We demonstrate that this frequency structure explains why shared-attention mechanisms, where a target image is generated while attending to tokens from a reference image, can lead to reference copying, in which the model reproduces content from the reference instead of extracting only its stylistic cues. Our analysis reveals that the high-frequency components of RoPE dominate the attention computation, forcing queries to attend mainly to spatially aligned reference tokens and thereby inducing this unintended copying behavior. Building on these insights, we introduce a method for selectively modulating RoPE frequency bands so that attention reflects semantic similarity rather than strict positional alignment. Applied to modern transformer-based diffusion architectures, where all tokens share attention, this modulation restores stable and meaningful shared attention. As a result, it enables effective control over the degree of style transfer versus content copying, yielding a proper style-aligned generation process in which stylistic attributes are transferred without duplicating reference content.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。