arXiv:2410.11439cs.CV2024-10ICLR被引 18

一个简单框架统一图像条件生成,支持多种控制方式。

A Simple Approach to Unifying Diffusion-based Conditional Generation

  • 用扩散模型学习图像对联合分布,推理时切换采样策略
  • 仅需15%额外参数,单次训练达成多任务生成与估计
  • 适合需要灵活控制的图像生成场景,如深度转图像

近期图像生成研究推动了通过条件信号控制模型的发展,但现有方法多针对特定问题。本文提出一种简洁统一的框架,通过扩散模型学习相关图像对(如图像与深度图)的联合分布,利用不同推理阶段采样策略实现可控图像生成、估计、信号引导、联合生成及粗略控制。相比以往统一方法依赖多阶段训练、架构修改或大量参数,本方法仅需一次高效训练,保持标准输入,新增参数仅占基础模型的15%。此外,支持非空间对齐和粗粒度条件输入。大量实验表明,单一模型性能可媲美专用方法,优于已有统一方案;多模型组合还可实现多信号条件生成。

原文摘要 · Abstract (English)

Recent progress in image generation has sparked research into controlling these models through condition signals, with various methods addressing specific challenges in conditional generation. Instead of proposing another specialized technique, we introduce a simple, unified framework to handle diverse conditional generation tasks involving a specific image-condition correlation. By learning a joint distribution over a correlated image pair (e.g. image and depth) with a diffusion model, our approach enables versatile capabilities via different inference-time sampling schemes, including controllable image generation (e.g. depth to image), estimation (e.g. image to depth), signal guidance, joint generation (image & depth), and coarse control. Previous attempts at unification often introduce significant complexity through multi-stage training, architectural modification, or increased parameter counts. In contrast, our simple formulation requires a single, computationally efficient training stage, maintains the standard model input, and adds minimal learned parameters (15% of the base model). Moreover, our model supports additional capabilities like non-spatially aligned and coarse conditioning. Extensive results show that our single model can produce comparable results with specialized methods and better results than prior unified methods. We also demonstrate that multiple models can be effectively combined for multi-signal conditional generation.

扩散模型条件生成图像修复

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。