UniCon通过单向信息流提升扩散模型控制效率
UniCon: Unidirectional Information Flow for Effective Control of Large-Scale Diffusion Models
- 控制适配器仅接收扩散网络输出，无需双向交互
- 训练速度提升2.3倍，显存占用减少三分之一
- 可训练参数量翻倍的适配器，适合高效可控生成
我们提出UniCon，一种新型架构，用于提升大规模扩散模型适配器训练的控制能力与效率。与现有依赖扩散模型与控制适配器双向交互的方法不同，UniCon采用从扩散网络到适配器的单向信息流，使适配器独立生成最终输出。该设计消除了训练时扩散模型计算和存储梯度的需求，显著降低计算开销。实验表明，UniCon将GPU显存使用减少三分之一，训练速度提升2.3倍，同时保持相同适配器参数量。此外，在不增加额外计算资源的前提下，支持训练参数量为现有ControlNets两倍的适配器。在一系列图像条件生成任务中，UniCon表现出对控制输入的精准响应和出色的生成能力。
原文摘要 · Abstract (English)
We introduce UniCon, a novel architecture designed to enhance control and efficiency in training adapters for large-scale diffusion models. Unlike existing methods that rely on bidirectional interaction between the diffusion model and control adapter, UniCon implements a unidirectional flow from the diffusion network to the adapter, allowing the adapter alone to generate the final output. UniCon reduces computational demands by eliminating the need for the diffusion model to compute and store gradients during adapter training. Our results indicate that UniCon reduces GPU memory usage by one-third and increases training speed by 2.3 times, while maintaining the same adapter parameter size. Additionally, without requiring extra computational resources, UniCon enables the training of adapters with double the parameter volume of existing ControlNets. In a series of image conditional generation tasks, UniCon has demonstrated precise responsiveness to control inputs and exceptional generation capabilities.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品；中文卡片由大模型生成，请以原文为准。