arXiv:2502.10451cs.LGcs.GR2025-02被引 3

让图像生成自动选关键步骤,省算力还更准。

FlexControl: Computation-Aware ControlNet with Differentiable Router for Text-to-Image Generation

  • 用可学习的门控机制动态决定每步激活哪些网络块。
  • 在SD1.5和SD3.0上提升生成质量,同时降低计算开销。
  • 适合想高效控制扩散模型生成过程的研究者和开发者。

ControlNet为基于扩散的生成模型提供了强大的控制能力,但现有实现多依赖人工启发式选择需控制的网络块,导致不同任务表现不一且不可预测。为此,我们提出FlexControl,一种新框架:训练时复制所有扩散模块,并引入可训练的门控机制,在每个去噪步骤动态选择激活哪些块。通过引入计算感知损失,仅当控制块有助于提升生成质量时才激活它们。该方法无需手动选择块,提升了跨任务适应性并简化设计流程,支持端到端训练。在UNet(如SD1.5)和DiT(如SD3.0)上的大量实验表明,本方法在多个关键指标上优于现有ControlNet变体。定量与定性评估均显示,FlexControl在保持或提升图像保真度的同时,通过选择性激活相关块降低了计算开销。结果证明,灵活的数据驱动控制策略对可控扩散模型具有潜力,并为高效生成模型设计开辟新路径。代码即将开源。

原文摘要 · Abstract (English)

ControlNet offers a powerful way to guide diffusion-based generative models, yet most implementations rely on ad-hoc heuristics to choose which network blocks to control-an approach that varies unpredictably with different tasks. To address this gap, we propose FlexControl, a novel framework that copies all diffusion blocks during training and employs a trainable gating mechanism to dynamically select which blocks to activate at each denoising step. With introducing a computation-aware loss, we can encourage control blocks only to activate when it benefit the generation quality. By eliminating manual block selection, FlexControl enhances adaptability across diverse tasks and streamlines the design pipeline, with computation-aware training loss in an end-to-end training manner. Through comprehensive experiments on both UNet (e.g., SD1.5) and DiT (e.g., SD3.0), we show that our method outperforms existing ControlNet variants in certain key aspects of interest. As evidenced by both quantitative and qualitative evaluations, FlexControl preserves or enhances image fidelity while also reducing computational overhead by selectively activating the most relevant blocks. These results underscore the potential of a flexible, data-driven approach for controlled diffusion and open new avenues for efficient generative model design. The code will soon be available at https://github.com/Anonymousuuser/FlexControl.

扩散模型图像生成控制生成高效推理

Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。