arXiv:2503.16153cs.CV2025-03ICCV被引 21

解析RoPE在图像生成中的分层作用,实现无需训练的精准编辑

FreeFlux: Understanding and Exploiting Layer-Specific Roles in RoPE-Based MMDiT for Versatile Image Editing

  • 通过可控干扰RoPE揭示各层位置与内容依赖差异
  • 提出三类任务适配的键值注入策略,显著提升编辑一致性
  • 适合需要精细控制生成内容的视觉编辑场景

将旋转位置编码(RoPE)引入多模态扩散变压器(MMDiT)显著提升了文生图生成质量。然而,自注意力层在生成过程中对位置编码与查询-键相似性的依赖关系仍不明确。本文首次对基于RoPE的MMDiT模型(如FLUX)进行机制分析,提出一种自动化探针策略,通过有策略地操控生成过程中的RoPE,分离位置信息与内容依赖。分析发现各层依赖模式并不简单随深度变化,为理解分层角色提供了新视角。基于此,我们提出一种无需训练、任务特异的图像编辑框架,将编辑任务分为三类:位置依赖型(如物体添加)、内容相似性依赖型(如非刚性变形)、区域保留型(如背景替换)。针对每类任务设计对应的键值注入策略。大量定性和定量评估表明,该方法优于现有最优方案,尤其在保持原始语义内容和实现无缝修改方面表现突出。

原文摘要 · Abstract (English)

The integration of Rotary Position Embedding (RoPE) in Multimodal Diffusion Transformer (MMDiT) has significantly enhanced text-to-image generation quality. However, the fundamental reliance of self-attention layers on positional embedding versus query-key similarity during generation remains an intriguing question. We present the first mechanistic analysis of RoPE-based MMDiT models (e.g., FLUX), introducing an automated probing strategy that disentangles positional information versus content dependencies by strategically manipulating RoPE during generation. Our analysis reveals distinct dependency patterns that do not straightforwardly correlate with depth, offering new insights into the layer-specific roles in RoPE-based MMDiT. Based on these findings, we propose a training-free, task-specific image editing framework that categorizes editing tasks into three types: position-dependent editing (e.g., object addition), content similarity-dependent editing (e.g., non-rigid editing), and region-preserved editing (e.g., background replacement). For each type, we design tailored key-value injection strategies based on the characteristics of the editing task. Extensive qualitative and quantitative evaluations demonstrate that our method outperforms state-of-the-art approaches, particularly in preserving original semantic content and achieving seamless modifications.

图像编辑RoPE扩散模型生成控制

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。