无需重训练,一键让扩散模型受控生成
Noise Consistency Training: A Native Approach for One-Step Generator in Learning Additional Controls
- 在噪声空间引入一致性损失,轻量适配新控制条件
- 单次前向传播即达最优可控生成效果,速度超越多步方法
- 适合需要快速部署可控生成的开发者和应用团队
高效且可控的高质量内容生成仍是AIGC的核心挑战。尽管基于扩散蒸馏的一步生成器在生成质量和计算效率上表现优异,但将其适配至结构约束、语义指导或外部输入等新控制条件仍面临重大困难。传统方法需对基础模型进行昂贵修改并重新蒸馏。本文提出噪声一致性训练(NCT),一种轻量级新方法,可在不访问原始训练图像或重训练扩散模型的前提下,直接将新控制信号注入预训练的一步生成器。NCT通过引入适配模块,并在生成器的噪声空间中使用噪声一致性损失,使不同条件依赖程度的噪声生成行为保持一致,隐式引导模型遵循新控制。该目标可理论解释为最小化适配生成器与新条件诱导的条件分布间的分布距离。NCT模块化、数据高效、易于部署,仅需预训练生成器和控制信号模型。大量实验表明,NCT在单次前向传播中实现最先进可控生成,在生成质量与计算效率上均优于现有多步及蒸馏方法。代码已开源。
原文摘要 · Abstract (English)
The pursuit of efficient and controllable high-quality content generation remains a central challenge in artificial intelligence-generated content (AIGC). While one-step generators, enabled by diffusion distillation techniques, offer excellent generation quality and computational efficiency, adapting them to new control conditions--such as structural constraints, semantic guidelines, or external inputs--poses a significant challenge. Conventional approaches often necessitate computationally expensive modifications to the base model and subsequent diffusion distillation. This paper introduces Noise Consistency Training (NCT), a novel and lightweight approach to directly integrate new control signals into pre-trained one-step generators without requiring access to original training images or retraining the base diffusion model. NCT operates by introducing an adapter module and employs a noise consistency loss in the noise space of the generator. This loss aligns the adapted model's generation behavior across noises that are conditionally dependent to varying degrees, implicitly guiding it to adhere to the new control. Theoretically, this training objective can be understood as minimizing the distributional distance between the adapted generator and the conditional distribution induced by the new conditions. NCT is modular, data-efficient, and easily deployable, relying only on the pre-trained one-step generator and a control signal model. Extensive experiments demonstrate that NCT achieves state-of-the-art controllable generation in a single forward pass, surpassing existing multi-step and distillation-based methods in both generation quality and computational efficiency. Code is available at https://github.com/Luo-Yihong/NCT
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。