arXiv:2410.01262cs.CVcs.LG2024-10

无需训练即可实现精细控制,通过融合多个扩散模型特征。

Improving Fine-Grained Control via Aggregation of Multiple Diffusion Models

  • 在潜在空间聚合多个扩散模型特征,实现无训练的精细生成控制。
  • 实验表明该方法显著提升细粒度控制能力,且不依赖复杂数据集或模型设计。
  • 适合希望快速集成现有条件生成模型的研究者与开发者使用。

尽管许多扩散模型在控制风格、角色和交互等特定方面表现良好,但受限于数据集和复杂的模型架构设计,在细粒度控制上仍面临挑战。本文提出一种全新的无训练算法——多扩散模型聚合(Aggregation of Multiple Diffusion Models, AMDM),将同一生态中多个扩散模型在潜在空间中的特征整合到指定模型中,从而激活特定属性,实现细粒度控制。实验结果表明,AMDM在不进行训练的情况下显著提升了细粒度控制效果,验证了其有效性。此外,研究发现扩散模型初期更关注位置、属性和风格等特征,后期才逐步提升生成质量和一致性。AMDM为解决扩散模型中细粒度条件生成难题提供了新思路:可充分复用现有或新开发的特定条件扩散模型,通过AMDM算法进行聚合,避免构建复杂数据集、设计复杂架构及高昂训练成本。代码已公开:https://github.com/Hammour-steak/AMDM。

原文摘要 · Abstract (English)

While many diffusion models perform well when controlling particular aspects such as style, character, and interaction, they struggle with fine-grained control due to dataset limitations and intricate model architecture design. This paper introduces a novel training-free algorithm for fine-grained generation, called Aggregation of Multiple Diffusion Models (AMDM). The algorithm integrates features in the latent data space from multiple diffusion models within the same ecosystem into a specified model, thereby activating particular features and enabling fine-grained control. Experimental results demonstrate that AMDM significantly improves fine-grained control without training, validating its effectiveness. Additionally, it reveals that diffusion models initially focus on features such as position, attributes, and style, with later stages improving generation quality and consistency. AMDM offers a new perspective for tackling the challenges of fine-grained conditional generation in diffusion models. Specifically, it allows us to fully utilize existing or develop new conditional diffusion models that control specific aspects, and then aggregate them using the AMDM algorithm. This eliminates the need for constructing complex datasets, designing intricate model architectures, and incurring high training costs. Code is available at: https://github.com/Hammour-steak/AMDM.

扩散模型细粒度控制无训练

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。