让扩散模型生成结果符合指定属性分布,无需重训练
Inference-Time Attribute Distribution Alignment for Unconditional Diffusion

- 将生成过程视为动态系统,用时变扰动控制分布
- 可灵活对齐多种属性目标,保持图像质量不下降
- 适合需要特定人群或语义比例的生成应用
推理时可控生成对无条件扩散模型的实际应用至关重要。然而,现有方法多关注单个样本,难以满足需群体属性分布匹配(如人口平衡或语义比例)的应用需求。本文将此问题形式化为预训练无条件扩散模型的推理时属性分布对齐问题。通过将逆扩散过程建模为动态系统,引入时变附加扰动作为控制项,将其转化为最优控制问题。采用基于最优控制的算法求解扰动,以优化可微的分布匹配目标,并惩罚控制代价以维持数据保真度。实验表明,该即插即用方法在图像生成任务中能更有效地对齐多样且灵活的测试时目标分布,优于基线方法,且无需重训练或微调预训练扩散模型。
原文摘要 · Abstract (English)
Inference-time controllable generation is essential for real-world applications of unconditional diffusion models. However, most existing techniques focus on individual samples, struggling in applications that require the sample population to follow specific attribute distributions (e.g., demographic balance or semantic proportions). We formalize this setting as the inference-time attribute distributional alignment problem for pretrained unconditional diffusion models. To address this, we cast inference-time attribute distributional alignment as an optimal control problem over the reverse diffusion process, viewing the process as the rollout of a dynamical system and augmenting it with additive, time-dependent perturbations as control. We solve for the perturbations using an optimal-control-based algorithm to optimize a differentiable distribution-matching objective while penalizing control effort to preserve data fidelity. Experiment results in image generation demonstrate that our proposed plug-and-play approach can better align attribute distributions to diverse and flexible test-time targets compared to baselines, without retraining or finetuning the pretrained diffusion model.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。