动态选择控制信号,提升文本生成图像的可控性与细节。
DynamicControl: Adaptive Condition Selection for Improved Text-to-Image Generation
- 通过双循环控制器和多模态大模型动态筛选条件
- 在COCO、ImageNet上实现更高图像质量和条件组合能力
- 适合需要精细控制的图像生成任务,如设计与艺术创作
为提升文本到图像扩散模型的可控性,现有ControlNet类方法尝试多种控制信号以调控图像属性。但现有方法或条件处理效率低,或固定使用条件数量,无法有效应对多条件复杂性和潜在冲突。为此,我们提出DynamicControl框架,支持多样化控制信号的动态组合,可自适应选择条件的数量与类型。方法首先通过双循环控制器,利用预训练条件生成模型和判别模型,对所有输入条件生成初始真实度评分排序,评估条件间相似性及与源图像的像素级相似性。随后引入多模态大语言模型(MLLM)构建高效条件评估器,基于双循环控制器的评分排序优化条件顺序。该方法联合优化MLLM与扩散模型,借助MLLM推理能力促进多条件文本到图像生成任务。最终排序后的条件输入并行多控制适配器,学习动态视觉条件特征图并融合,调制ControlNet以增强生成图像的控制力。定量与定性实验表明,DynamicControl在多种条件控制下均优于现有方法,在可控性、生成质量与组合性方面表现更优。
原文摘要 · Abstract (English)
To enhance the controllability of text-to-image diffusion models, current ControlNet-like models have explored various control signals to dictate image attributes. However, existing methods either handle conditions inefficiently or use a fixed number of conditions, which does not fully address the complexity of multiple conditions and their potential conflicts. This underscores the need for innovative approaches to manage multiple conditions effectively for more reliable and detailed image synthesis. To address this issue, we propose a novel framework, DynamicControl, which supports dynamic combinations of diverse control signals, allowing adaptive selection of different numbers and types of conditions. Our approach begins with a double-cycle controller that generates an initial real score sorting for all input conditions by leveraging pre-trained conditional generation models and discriminative models. This controller evaluates the similarity between extracted conditions and input conditions, as well as the pixel-level similarity with the source image. Then, we integrate a Multimodal Large Language Model (MLLM) to build an efficient condition evaluator. This evaluator optimizes the ordering of conditions based on the double-cycle controller's score ranking. Our method jointly optimizes MLLMs and diffusion models, utilizing MLLMs' reasoning capabilities to facilitate multi-condition text-to-image (T2I) tasks. The final sorted conditions are fed into a parallel multi-control adapter, which learns feature maps from dynamic visual conditions and integrates them to modulate ControlNet, thereby enhancing control over generated images. Through both quantitative and qualitative comparisons, DynamicControl demonstrates its superiority over existing methods in terms of controllability, generation quality and composability under various conditional controls.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。