让文生图模型精准控制图像布局，支持边缘、姿态等多种条件。
Adding Conditional Control to Text-to-Image Diffusion Models
- 通过零初始化卷积层，在不破坏原模型的前提下添加空间控制
- 在小至5万、大至百万级数据上训练均稳定有效
- 适合需要精确控制生成图像结构的设计师与开发者
我们提出ControlNet，一种神经网络架构，用于向大型预训练文生图扩散模型添加空间条件控制。ControlNet锁定已部署的大规模扩散模型，复用其经过数十亿图像预训练的深层编码层作为强大骨干，学习多种条件控制。神经架构采用“零卷积”（零初始化卷积层），逐步从零增长参数，确保微调过程不受有害噪声干扰。我们在Stable Diffusion上测试了边缘、深度、分割、人体姿态等多种条件控制，支持单条件或多条件输入，有无提示词均可。实验表明，ControlNet在小样本（<5万）和大规模（>100万）数据集上训练均具鲁棒性。大量结果证明，ControlNet可显著拓展文生图模型的应用范围。
原文摘要 · Abstract (English)
We present ControlNet, a neural network architecture to add spatial conditioning controls to large, pretrained text-to-image diffusion models. ControlNet locks the production-ready large diffusion models, and reuses their deep and robust encoding layers pretrained with billions of images as a strong backbone to learn a diverse set of conditional controls. The neural architecture is connected with "zero convolutions" (zero-initialized convolution layers) that progressively grow the parameters from zero and ensure that no harmful noise could affect the finetuning. We test various conditioning controls, eg, edges, depth, segmentation, human pose, etc, with Stable Diffusion, using single or multiple conditions, with or without prompts. We show that the training of ControlNets is robust with small (<50k) and large (>1m) datasets. Extensive results show that ControlNet may facilitate wider applications to control image diffusion models.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。