无需训练即可调节扩散模型注意力频谱,实现精细图像生成控制。
Attention Frequency Modulation: Training-Free Spectral Modulation of Diffusion Cross-Attention
- 在傅里叶域重加权交叉注意力,实现推理时频域调控。
- 可连续调整注意力空间尺度,显著改变生成图像但保持语义一致。
- 基于注意力熵自适应调节,适合需要精准控制的生成任务。
交叉注意力是文本条件潜空间扩散模型的核心接口,但其逐步多分辨率动态特性尚未充分理解,限制了无需训练的可控生成。本文将扩散交叉注意力视为潜在网格上的时空信号,通过汇总令牌-软最大权重生成与令牌无关的集中度图,并追踪去噪过程中径向分组的傅里叶功率。在不同提示和随机种子下,编码器交叉注意力呈现出一致的粗到细频谱演化,形成稳定的令牌竞争时间-频率指纹。基于此结构,我们提出注意力频谱调制(AFM),一种即插即用的推理时干预方法:在软最大前,对令牌级预软最大交叉注意力逻辑进行傅里叶域重加权,低频和高频分量按进度对齐调度,并可由令牌分配熵自适应门控。AFM 提供了无需重训练、提示编辑或参数更新的连续空间尺度偏置控制。在 Stable Diffusion 上的实验表明,AFM 能可靠地重新分布注意力频谱,产生显著视觉修改,同时基本保持语义一致性。最后发现,熵主要作为相同频谱编辑的自适应增益,而非独立控制轴。
原文摘要 · Abstract (English)
Cross-attention is the primary interface through which text conditions latent diffusion models, yet its step-wise multi-resolution dynamics remain under-characterized, limiting principled training-free control. We cast diffusion cross-attention as a spatiotemporal signal on the latent grid by summarizing token-softmax weights into token-agnostic concentration maps and tracking their radially binned Fourier power over denoising. Across prompts and seeds, encoder cross-attention exhibits a consistent coarse-to-fine spectral progression, yielding a stable time-frequency fingerprint of token competition. Building on this structure, we introduce Attention Frequency Modulation (AFM), a plug-and-play inference-time intervention that edits token-wise pre-softmax cross-attention logits in the Fourier domain: low- and high-frequency bands are reweighted with a progress-aligned schedule and can be adaptively gated by token-allocation entropy, before the token softmax. AFM provides a continuous handle to bias the spatial scale of token-competition patterns without retraining, prompt editing, or parameter updates. Experiments on Stable Diffusion show that AFM reliably redistributes attention spectra and produces substantial visual edits while largely preserving semantic alignment. Finally, we find that entropy mainly acts as an adaptive gain on the same frequency-based edit rather than an independent control axis.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。