轻量级控制框架,让扩散Transformer生成更精准且高效。
NanoControl: A Lightweight Framework for Precise and Efficient Control in Diffusion Transformer
- 用低秩适配模块直接学习条件信号,不复制主干网络
- 参数仅增0.024%,计算量增0.029%,性能领先
- 适合追求高效可控图像生成的研究与应用
扩散Transformer(DiTs)在文本到图像生成中表现卓越。然而,在可控文本到图像生成领域,现有方法大多沿用为UNet设计的ControlNet范式,带来显著参数开销和计算成本。为此,我们提出轻量级控制框架NanoControl,以Flux为骨干网络。该模型在仅增加0.024%参数量和0.029% GFLOPs的前提下,实现顶尖的可控图像生成性能。不同于复制DiT主干,我们设计了类似LoRA的控制模块,直接从原始条件输入中学习控制信号。此外,引入KV-Context Augmentation机制,以简单有效方式将特定条件的键值信息融入主干,促进条件特征深层融合。大量基准实验表明,相比传统控制方法,NanoControl大幅降低计算开销,同时保持优异生成质量并提升可控性。
原文摘要 · Abstract (English)
Diffusion Transformers (DiTs) have demonstrated exceptional capabilities in text-to-image synthesis. However, in the domain of controllable text-to-image generation using DiTs, most existing methods still rely on the ControlNet paradigm originally designed for UNet-based diffusion models. This paradigm introduces significant parameter overhead and increased computational costs. To address these challenges, we propose the Nano Control Diffusion Transformer (NanoControl), which employs Flux as the backbone network. Our model achieves state-of-the-art controllable text-to-image generation performance while incurring only a 0.024\% increase in parameter count and a 0.029\% increase in GFLOPs, thus enabling highly efficient controllable generation. Specifically, rather than duplicating the DiT backbone for control, we design a LoRA-style (low-rank adaptation) control module that directly learns control signals from raw conditioning inputs. Furthermore, we introduce a KV-Context Augmentation mechanism that integrates condition-specific key-value information into the backbone in a simple yet highly effective manner, facilitating deep fusion of conditional features. Extensive benchmark experiments demonstrate that NanoControl significantly reduces computational overhead compared to conventional control approaches, while maintaining superior generation quality and achieving improved controllability.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。