统一框架实现多条件图像生成,效率与可控性兼备。
Condition Weaving Meets Expert Modulation: Towards Universal and Controllable Image Generation
- 用专家模块按语义分组特征,避免多条件下的冗余计算。
- 在两个数据集上均达当前最佳性能,支持多种生成任务。
- 适合需要高效多条件控制的图像生成研究者使用。
图像到图像生成旨在通过条件输入和提示指令生成可控图像。现有方法通常为每种条件训练独立的控制分支,导致模型结构冗余、计算资源利用低效。为此,我们提出统一的图像生成框架UniGen,支持多样条件输入,提升生成效率与表达力。针对条件生成架构中普遍存在的参数冗余与计算低效问题,提出条件调制专家(CoMoE)模块:聚合语义相似的局部特征,并分配给专用专家模块进行视觉表征与条件建模;通过在不同条件下独立建模前景特征,有效缓解多条件场景中的特征混淆与冗余计算。此外,为弥合主干网络与控制分支间的信息鸿沟,提出WeaveNet——一种动态蛇形连接机制,实现主干网络提供的全局文本级控制与条件分支的细粒度控制之间的高效交互。在Subjects-200K和MultiGen-20M两个数据集上,涵盖多种条件图像生成任务的大量实验表明,该方法持续达到最先进性能,验证了其在通用性与有效性方面的优势。代码已公开于https://github.com/gavin-gqzhang/UniGen。
原文摘要 · Abstract (English)
The image-to-image generation task aims to produce controllable images by leveraging conditional inputs and prompt instructions. However, existing methods often train separate control branches for each type of condition, leading to redundant model structures and inefficient use of computational resources. To address this, we propose a Unified image-to-image Generation (UniGen) framework that supports diverse conditional inputs while enhancing generation efficiency and expressiveness. Specifically, to tackle the widely existing parameter redundancy and computational inefficiency in controllable conditional generation architectures, we propose the Condition Modulated Expert (CoMoE) module. This module aggregates semantically similar patch features and assigns them to dedicated expert modules for visual representation and conditional modeling. By enabling independent modeling of foreground features under different conditions, CoMoE effectively mitigates feature entanglement and redundant computation in multi-condition scenarios. Furthermore, to bridge the information gap between the backbone and control branches, we propose WeaveNet, a dynamic, snake-like connection mechanism that enables effective interaction between global text-level control from the backbone and fine-grained control from conditional branches. Extensive experiments on the Subjects-200K and MultiGen-20M datasets across various conditional image generation tasks demonstrate that our method consistently achieves state-of-the-art performance, validating its advantages in both versatility and effectiveness. The code has been uploaded to https://github.com/gavin-gqzhang/UniGen.
Thank you to arXiv for use of its open access interoperability. PaperDance 不是 arXiv 官方产品;中文卡片由大模型生成,请以原文为准。