无需训练即可同时精准融合新物体和新风格,提升扩散模型编辑效果。
TP-Blend: Textual-Prompt Attention Pairing for Precise Object-Style Blending in Diffusion Models
- 通过双注意力机制分别处理物体与风格信息,实现独立融合。
- 在SD-XL上实现高分辨率真实感编辑,速度优于现有方法。
- 适合需要精细控制内容与外观的图像编辑用户。
当前文本条件扩散编辑器在单一物体替换上表现良好,但在同时引入新物体和新风格时表现不佳。本文提出无需训练的TP-Blend框架,接收两个独立文本提示:一个指定融合物体,另一个定义目标风格,并将其注入单一去噪轨迹。该框架由两个互补注意力处理器驱动:跨注意力物体融合(CAOF)首先平均各头注意力以定位对任一提示响应强的空间位置,再通过熵正则化最优传输问题将完整多头特征向量重分配至这些位置;其在全维度(如SD-XL中640维)更新特征向量,保留丰富跨头相关性且内存开销低。自注意力风格融合(SASF)在每层自注意力中通过细节敏感实例归一化注入风格:轻量一维高斯滤波分离高低频成分,仅将高频残差融合回原图,实现笔触级纹理刻画而不破坏整体几何结构;此外,用风格提示导出的键值矩阵替换原键值矩阵,实现上下文感知的纹理调制,且与物体融合解耦。大量实验表明,TP-Blend在高分辨率、照片级真实感编辑中均表现优异,在保真度、感知质量与推理速度上超越近期基线。
原文摘要 · Abstract (English)
Current text-conditioned diffusion editors handle single object replacement well but struggle when a new object and a new style must be introduced simultaneously. We present Twin-Prompt Attention Blend (TP-Blend), a lightweight training-free framework that receives two separate textual prompts, one specifying a blend object and the other defining a target style, and injects both into a single denoising trajectory. TP-Blend is driven by two complementary attention processors. Cross-Attention Object Fusion (CAOF) first averages head-wise attention to locate spatial tokens that respond strongly to either prompt, then solves an entropy-regularised optimal transport problem that reassigns complete multi-head feature vectors to those positions. CAOF updates feature vectors at the full combined dimensionality of all heads (e.g., 640 dimensions in SD-XL), preserving rich cross-head correlations while keeping memory low. Self-Attention Style Fusion (SASF) injects style at every self-attention layer through Detail-Sensitive Instance Normalization. A lightweight one-dimensional Gaussian filter separates low- and high-frequency components; only the high-frequency residual is blended back, imprinting brush-stroke-level texture without disrupting global geometry. SASF further swaps the Key and Value matrices with those derived from the style prompt, enforcing context-aware texture modulation that remains independent of object fusion. Extensive experiments show that TP-Blend produces high-resolution, photo-realistic edits with precise control over both content and appearance, surpassing recent baselines in quantitative fidelity, perceptual quality, and inference speed.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。