用少量数据快速训练可控图像生成模型，降低使用门槛。
CtrLoRA: An Extensible and Efficient Framework for Controllable Image Generation
- 先训练通用控制网络，再用轻量LoRA适配特定条件。
- 仅需1000组数据、不到1小时单卡训练即可达到良好效果。
- 参数量减少90%，适合普通用户快速部署新控制条件。
近期大规模扩散模型在文本到图像生成方面取得显著进展。为实现更精细的空间控制，ControlNet等方法引入额外网络以学习条件图像的映射关系。然而，每种条件类型都需要在数百万对数据上独立训练，耗时数百GPU小时，成本高昂，普通用户难以探索新条件。为此，我们提出CtrLoRA框架：先训练一个基础控制网络（Base ControlNet），从多种基础条件中学习图像到图像生成的通用知识，并结合特定条件的LoRA模块捕捉各条件的独特特征。利用预训练的Base ControlNet，用户可仅用1000组数据和不足一小时的单卡训练，即在多数场景下获得满意结果。此外，相比ControlNet，CtrLoRA将可学习参数减少90%，显著降低模型分发与部署门槛。在多种条件类型上的大量实验验证了该方法的高效性与有效性。代码与模型权重将公开于https://github.com/xyfJASON/ctrlora。
原文摘要 · Abstract (English)
Recently, large-scale diffusion models have made impressive progress in text-to-image (T2I) generation. To further equip these T2I models with fine-grained spatial control, approaches like ControlNet introduce an extra network that learns to follow a condition image. However, for every single condition type, ControlNet requires independent training on millions of data pairs with hundreds of GPU hours, which is quite expensive and makes it challenging for ordinary users to explore and develop new types of conditions. To address this problem, we propose the CtrLoRA framework, which trains a Base ControlNet to learn the common knowledge of image-to-image generation from multiple base conditions, along with condition-specific LoRAs to capture distinct characteristics of each condition. Utilizing our pretrained Base ControlNet, users can easily adapt it to new conditions, requiring as few as 1,000 data pairs and less than one hour of single-GPU training to obtain satisfactory results in most scenarios. Moreover, our CtrLoRA reduces the learnable parameters by 90% compared to ControlNet, significantly lowering the threshold to distribute and deploy the model weights. Extensive experiments on various types of conditions demonstrate the efficiency and effectiveness of our method. Codes and model weights will be released at https://github.com/xyfJASON/ctrlora.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。