无需训练即可精准控制图像生成，提升细节质量与一致性
Concept Guidance: Precise, Training-Free Latent Control for Text-to-Image Generation
- 基于概念互信息识别关键网络层，实现目标特定引导
- 跳过相关层生成加权预测，显著提升文本与手部等局部结构质量
- 即插即用，兼容PixArt-alpha、SD3等主流模型，无需额外参数
文本到图像扩散模型存在两大缺陷：一是缺乏连续、概念特定的引导机制（如精确控制图像美学质量），二是对高局部一致性任务（如生成文字或人手）可靠性不足。为此，我们提出概念级互信息新概念，发现不同层在特定结构生成中存在显著差异，表明特定结构的生成集中在网络不同区域。基于此，我们设计了无需训练的Concept Guidance（CoG）方法，通过量化各层的概念相关性，并在去噪过程中使用跳过相关层后的预测加权组合进行引导。该方法无需额外训练、外部模型、梯度计算或提示工程，可直接应用于PixArt-alpha、SD3、SD3.5、FLUX.1-dev等主流模型，在多种目标上均实现性能提升。代码已开源。
原文摘要 · Abstract (English)
Text-to-image diffusion models have two major drawbacks that severely limit their practical utility: (1) standard models lack an intrinsic mechanism for continuous, concept-specific guidance (e.g., for precisely controlling how aesthetically pleasing an image looks), and (2) they lack reliability for tasks requiring high local coherence (e.g., generating text or human hands). To tackle these issues, we introduce a novel notion of concept-wise mutual information and find large, concept-dependent differences between individual layers, demonstrating that the generation of specific structures is localized in distinct parts of the network. We exploit this insight by reinforcing the impact of concept-relevant layers in Concept Guidance (CoG), a precise, target-specific guidance method that works for models out-of-the-box without additional training, external models, gradients, or prompt engineering. CoG first quantifies each layer's concept-specific impact and then guides denoising using a weighted combination of predictions generated with concept-relevant layers skipped. We demonstrate performance increases across various targets and popular models like PixArt-alpha, SD3, SD3.5, and FLUX.1-dev. Code is available at https://github.com/CompVis/concept_guidance
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。